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The  Future  Battlefield  Conditions  (FBC)  Team  of  the  Armored  Forces  Research  Unit,  U.S 
Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  has  a  Science  and 
Technology  Objective  (STO)  entitled  “Force  XXI  Training  Strategies.”  This  STO  is  also 
reflected  in  the  FBC  work  package  (2228)  FASTRAIN:  Force  XXI  Training  Methods  and 
Strategies.  Recent  work  under  this  work  package  has  involved  research  and  development 
concerning  training  for  digital  staffs.  In  order  to  continue  this  work,  a  contract  entitled 
“Prototype  Methods  for  the  Design  and  Evaluation  of  Training  and  Assessment  of  Digital  Staffs 
and  Crewmen”  was  issued.  The  major  purpose  of  this  effort  was  to  design,  develop  and 
implement  prototype  training  and  assessment  techmejues  for  future,  information  age,  staffs. 

This  report  concerns  research  focused  on  training  and  assessment  for  future  staffs.  It 
describes  an  examination  of  theories  of  team  training  applicable  to  information  age  staffs.  The 
report  documents  the  design  and  development  of  prototype  training  and  assessment  methods 
based  on  those  theories.  The  report  examines  implementation  of  the  protot)^®  techmques  in  a 
battalion  level  Battle  Command  Reengineering  (BCR)  III  experiment,  conducted  by  the  Mounted 
Maneuver  Battlespace  Lab  located  at  Fort  Knox,  KY.  Lessons  learned  from  both  the  examination 
and  implementation  of  these  theories  of  team  training  are  also  discussed. 

At  least  two  major  audiences  may  be  interested  in  this  report.  Researchers  interested  in 
the  area  of  training  for  information  age  staffs  will  find  an  examination  of  relevant  team  training 
theories,  and  prototype  training  and  assessment  methods  rooted  in  those  theories.  Also,  the 
report  may  be  of  interest  to  training  developers,  in  that  it  describes  an  effort  to  implement  digital 
staff  training  in  a  specific  context  (BCR  111).  Thus  this  report  may  prove  useful  in  future  research 
and  development  efforts  for  training  of  information  age  staffs. 

The  prototype  products  developed  under  this  effort  are  documented  in  a  five-volume  set 
of  materials  entitled  Trainini^  and  Measurement  Support  Package,  Mounted  Maneuver 
Battlespace  Lab,  Battle  Command  Reengineering,  available  from  the  Mounted  Maneuver 
Battlespace  Lab  (MMBL).  Training  and  evaluation  findings  from  this  effort  are  included  in  the 
MMBL's  Battle  Lab  Experiment  Final  Report  (BLEFR)  for  Battle  Command  Reengineering, 
Phase  III  (\999). 

The  research  reflected  in  this  report  was  briefed  to  sponsors  throughout  the  effort  and  in  a 
final  In  Progress  Review,  held  at  Armored  Forces  Research  Unit,  Fort  Knox,  KY,  on  8  June 

1999.  ^  /)  n  .  . 

ZWA  M.  SIMUTIS 
Technical  Director 
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EXECUTIVE  SUMMARY 


Research  Requirement: 

The  U.  S.  Army  is  currently  developing  and  fielding  information  systems  for  the  digital 
battlefield  of  the  future.  In  support  of  this  effort,  the  U.  S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences  (ARI),  Armored  Forces  Research  Unit,  Future  Battlefield 
Conditions  Team  is  engaged  in  the  design  and  development  of  training  and  performance 
evaluation  techniques.  For  this  project,  ARI’s  objective  was  to  design  team  training  and 
assessment  strategies  for  staff  operations  in  the  future  digital  tactical  operations  center  at  brigade 
and  below.  In  order  to  accomplish  this  objective,  the  team  designed  and  developed  two  primary 
products:  a  prototype  training  package  to  improve  staff  performance  and  a  prototype  evaluation 
package. 

The  prototype  products  were  implemented  during  the  Battle  Command  Reengineering 
(BCR)  in  Concept  Experimentation  Program  (CEP)  experiment  at  the  Mounted  Maneuver 
Battlespace  Lab  (MMBL)  at  Fort  Knox.  The  ARI’s  purpose  for  participating  in  this  experiment 
was  to  gain  additional  information  on  future  staff  training  requirements,  gather  feedback  for 
improvements  to  the  prototype  training  package,  and  gain  experience  with  specific  automated 
measures  of  performance  and  effectiveness  in  a  virtual  simulation  environment. 

Procedure: 

Previous  ARI  research  on  stmctured  training  programs  provided  the  model  for  the 
development  of  the  training  support  packages.  The  project  team  also  reviewed  available  literature 
regarding  training,  team  performance,  and  measurement  issues.  This  review  included  study  of 
the  U.  S.  Army  Training  and  Doctrine  Coimnand  (TRADOC)  Digital  Learning  Strategy 
(Department  of  the  Army,  1998)  and  associated  training  strategies  and  approaches,  professional 
psychological  publications,  training  and  education  literature,  tactical  operations,  and  command 
and  control  operations.  This  literature  review  provided  the  basis  for  decisions  concerning 
training  stractures  and  content,  measures  design,  and  analysis  procedures. 

An  extensive  ftont-end  analysis  was  conducted  to  define  future  battalion-level  staff 
processes  in  terms  of  individual  and  node  responsibilities,  specific  nodes  tasks  and  requirements, 
command,  control,  communications,  computer,  and  intelligence  (C^I)  systems  capabilities,  and 
individual  functions  and  tasks  using  C^I  tools.  After  analysis,  structured  training  exercises  were 
designed  and  developed  that  combined  O'*!  system  practice  with  techniques  to  focus  team 
training  on  shared  situational  awareness,  roles  and  functions,  and  decision-making  processes. 

The  approach  to  performance  evaluation  was  mviltifaceted,  combining  automated 
measures,  surveys,  observations,  and  interviews.  The  measurements  focused  on  various  issues 


concerning  command  and  control,  team  processes  and  decision-making,  team  roles  and 
responsibilities,  workload,  and  communication. 

The  prototype  training  and  evaluation  packages  were  partially  implemented  during  the 
BCR  m  experiment,  which  took  place  12  April  through  30  April  1999.  The  experiment  was 
conducted  in  the  MMBL  test  bed  at  Fort  Knox  with  the  2"*^  Squadron,  2"'*  Armored  Cavalry 
Regiment  from  Fort  Polk  participating.  The  major  research  products  associated  with  training  and 
evaluation  for  the  MMBL  implementation  are  presented  in  the  five-volume  set  of  materials 
entitled  Training  and  Measurement  Support  Package,  Battle  Command  Reengineering  III, 
Mounted  Maneuver  Battlespace  Lab  (Mounted  Maneuver  Battlespace  Lab,  1999). 

Findings: 

During  the  trial  implementation  of  the  prototype  training  package,  the  challenges  and 
stresses  associated  with  learning  the  new  organization,  equipment,  tactics,  and  C  I  system 
overrode  the  team  training  sessions.  A  major  finding  was  that  additional  analysis  is  required  to 
determine  if  this  type  of  training  should  be  done  in  conjunction  with  learning  to  operate  an 
advanced  C**!  system  or  if  these  sessions  should  be  implemented  in  a  less  demanding  operational 
environment  where  the  benefits  of  incorporating  them  into  staff  training  (which  are  described  in 
the  theoretical  literature  and  were  recognized  by  some  of  the  training  participants)  can  be  fully 
realized.  Additional  findings  related  to  team  training  sessions  include:  a)  using  a  facilitator  from 
outside  of  the  unit  to  implement  these  sessions;  b)  revising  the  information  management  training 
session;  and  c)  revising  the  titles  of  the  team  training  sessions  to  reflect  current  Army 
terminology.  In  findings  related  specifically  to  the  prototype  training  package,  additional 
refinement  of  the  training  design  is  needed  in  three  areas:  organization,  time  allocation,  and 
training  audience. 

Considerable  research  remains  to  be  done  on  developing  both  performance  standards  and 
evaluation  methods  for  future  battle  staffs  operating  advanced  C^I  systems.  While  surveys, 
observations,  and  interviews  were  used  to  gather  data  during  the  pilot  implementation,  the 
project  team  made  a  major  effort  to  develop  automated  measures  that  could  take  advantage  of  the 
analytical  power  and  processing  speed  of  advanced  C^I  systems  to  provide  real  or  near  real-time 
feedback  to  the  training  participants.  This  effort  was  only  partially  successful.  Further  research 
is  required  to  determine  the  specific  measures  and  processes  needed  to  extract  data  from 
advanced  C*!  systems  or  simulations  in  an  easily  interpretable  format  suitable  for  performance 
feedback. 

Utilization  of  Findings: 

These  findings  provide  insights  regarding  the  impacts  of  future  C'*!  systems  on  the  Battle 
Command  process.  The  specific  audience  who  may  find  the  mformation  contained  in  this  report 
beneficial  includes:  a)  training  program  designers,  developers,  and  implementers;  b)  simulation 
system  developers  (hardware  and  software);  and  c)  training  unit  and  training  site  personnel. 
Training  and  evaluation  findings  from  this  effort  are  included  in  the  MMBL’s  Battle  Lab 
Experiment  Final  Report  (BLEFR)for  Battle  Command  Reengineering,  Phase  III  (Mounted 
Maneuver  Battlespace  Lab,  1999b). 
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Introduction 

The  transition  to  the  digital  Army  of  Force  XXI  and  beyond  is  characterized  by 
challenges  to  how  the  Army  will  train,  maintain,  and  operate  as  an  information  age  force.  The 
future  Army  will  have  an  increased  area  of  operation;  combined  arms  for  close  combat  with 
fewer  systems;  non-linear,  asymmetrical  operations;  and  operational  requirements  for  situational 
awareness.  The  future  force  will  depend  on  the  capabilities  of  its  command,  control, 
communication,  computer,  and  intelligence  (O'*!)  systems,  and  on  the  ability  of  the  commanders 
and  staffs  to  fully  understand  and  utilize  the  systems. 

The  battalion  and  brigade  staff  of  the  future  Army  may  well  be  a  considerably  different 
entity  from  that  which  is  known  today.  Staffs  operate  on  information.  By  radically  increasing 
the  amount  and  timeliness  of  information  received  by  the  staff,  the  functions,  organization, 
capabilities,  and  requirements  of  the  staff  will  also  have  to  change.  But  staffs  are  complex 
organizations,  and  it  cannot  be  presumed  that  simply  adding  computers  and  digital 
communications  capabilities  will  provide  simple  answers  to  how  a  future  staff  will  be  organized 
and  what  tasks  it  will  perform. 

With  the  advanced  systems,  information  processing  takes  on  new  importance. 
Information  processing  is  a  component  of  decision-making  in  every  aspect  of  planning, 
preparation,  execution,  and  reconstitution.  As  they  make  decisions,  leaders  are  continuously 
dealing  with  updates  to  staff  estimates,  enemy  situations,  and  higher  headquarters  directives. 
Digital  information  systems  will  almost  certainly  increase  the  amount  and  complexity  of 
information  provided  to  the  staff,  and  training  should  provide  the  higher-order  skills  needed  on 
the  digital  battlefield.  “Better  education  and  training,  devoted  to  information  processing  under 
stress  and  in  environments  characterized  by  uncertainty,  are  needed  to  develop  the  necessary 
skills  to  handle  these  information-rich  situations”  (Alberts,  1996,  p.  32). 

There  are  a  number  of  concepts  in  the  current  and  recent  research  literature  related  to 
“information-rich  situations”  and  the  challenges  they  pose.  In  addition  to  the  problem  of 
information  management,  researchers  have  been  examining  the  impact  of  increased  information 
on  decision-making,  situational  awareness,  and  team  behaviors.  Within  the  various  models  that 
are  being  developed  to  explain  how  experts  and  teams  might  handle  information-rich  situations, 
there  are  also  indications  of  performance  techniques,  ways  to  train  and  educate,  and  ways  to 
ascertain  skill  acquisition  and  proficient  performance.  This  last  issue,  concerning  performance 
evaluation,  is  critical  to  training,  but  is  often  overlooked.  Evaluation  that  supports  both  training 
analysis  and  feedback  mechanisms  for  the  participants  is  needed. 

In  response  to  the  concerns  and  issues  resulting  from  digitization,  the  U.  S.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI),  Armored  Forces  Research  Unit, 
is  engaged  in  the  design  and  development  of  training  and  performance  evaluation  techniques  to 
support  Force  XXI  digital  capabilities.  The  ARI’s  research  in  this  area  includes  recent  advances 
in  the  cognitive  and  behavioral  sciences,  and  is  focused  on  providing  an  empirical  foundation  for 
improved  team  training  and  evaluation  strategies  for  the  digital  battlefield  of  the  future. 
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This  report  details  work  performed  in  support  of  that  objective  by  the  contractor 
consortium  of  the  Human  Resources  Research  Organization  (HumRRO),  Litton-PRC,  Klein 
Associates,  and  Aptima.  Specifically,  the  objective  of  the  research  project  was  to  design  team 
training  and  evaluation  strategies  for  staff  operations  in  the  future  digital  tactical  operations 
center  at  brigade  and  below.  The  research  addresses  methods  for  training  leaders  and  staffs  of 
future  digital  environments:  the  content  of  such  training,  the  methods  for  providing  the  training, 
and  the  means  for  assessing  performance  outcomes  as  a  result  of  training. 

The  work  began  with  a  review  of  research  literature  and  technical  documentation  related 
to  team  performance  and  training,  operations  in  digital  environments,  and  automated 
performance  data  collection.  A  general  design  for  staff  training  was  formulated,  based  on 
synthesis  of  the  literature  review.  This  design  was  then  tailored  to  provide  training  in  support  of 
an  upcoming  Army  experiment,  the  Battle  Command  Reengineering  (BCR)  HI,  which  took  place 
in  April  1999.  By  participating  in  the  BCR  in,  researchers  had  the  opportunity  to  conduct  a  trial 
implementation  of  the  training  and  evaluation  package.  Coordination  between  ARI  and  the 
Mounted  Maneuver  Battlespace  Lab  (MMBL)  at  Fort  Knox,  Kentucky  enabled  the  two 
organizations  to  work  together  as  a  team  to  accomplish  multiple  goals.  This  report  describes  the 
development  work  and  the  prototype  products,  results  of  their  use  during  the  BCR  HI,  and 
implications  for  future  training,  both  within  MMBL  settings  and  in  the  larger  context  of  staff 
training. 


Organization  of  the  Report 
This  report  has  five  sections: 

•  Introduction:  Summary  of  previous  research  and  relevant  literature  on  training  and 
evaluation  designs. 

•  Prototype  Training  and  Evaluation  Methods:  Description  of  the  front-end  analysis,  design, 
and  products  for  future  staff  training  and  evaluation;  description  of  the  BCR  HI  for  which 
prototype  development  and  trial  implementation  were  conducted. 

•  Formative  Results:  Description  of  the  BCR  HI  implementation;  results  and  discussion 
concerning  training  and  measures  evaluation. 

•  Lessons  Learned  for  Future  Research:  Summary  of  the  major  lessons  learned  concerning 
training  and  evaluation  with  implications  for  future  MMBL  experimentation  and  further 
research. 

•  Summary  and  Conclusions:  Brief  review  of  the  project’s  objectives  and  accomplishments. 

Appendix  A  contains  a  listing  of  the  acronyms  and  abbreviations  used  in  this  report. 
Samples  of  training  and  evaluation  products  developed  for  the  prototype  training  package  and 
trial  implementation  are  contained  in  Appendixes  B  through  F.  In  addition  to  this  report,  the 
major  research  products  associated  with  training  and  evaluation  for  the  MMBL  implementation 
are  presented  in  the  five-volume  set  of  materials  entitled  Training  and  Measurement  Support 
Package,  Battle  Command  Reengineering  III,  Mounted  Maneuver  Battlespace  Lab  (Mounted 
Maneuver  Battlespace  Lab,  1999c).  The  five  volumes  are: 
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•  Volume  1.  Front-End  Analysis  for  Training  and  Measurement.  Contains  all  of  the  front- 
end  analysis  products  developed  for  the  prototype  training  package  for  BCR  HI. 

•  Volume  2.  Initial  Orientation  and  Train-Up.  Includes  sample  slides  prepared  for  the 
MMBL  briefing  to  the  participating  unit  in  BCR  IQ,  training  plan  outlines  for  initial  and 
advanced  digital  system  training,  and  materials  for  the  first  two  team  training  sessions. 

•  Volume  3.  Tactical  Decision-Making  Exercises  and  Team  Training  Sessions.  Contains 
the  training  support  package  (TSP)  materials  for  the  structured  training  exercises  and 
team  training  developed  as  a  prototype  for  use  in  BCR  in. 

•  Volume  4.  Measures.  Includes  copies  of  all  surveys  and  structured  interview  forms, 
descriptions  and  screen  copies  from  the  observer  data  collection  instmment,  and 
specifications  for  the  automated  measures  used  in  BCR  HI. 

•  Volume  5.  Data  Codebook.  Contains  basic  descriptions  and  analyses  of  the  variables  in 
the  data  sets  resulting  from  the  data  collection  in  BCR  HI. 

Background 

The  purpose  of  this  section  is  to  provide  the  context  for  the  project  and  the  rationale  for 
the  approach  to  training  and  evaluation.  It  summarizes  the  research  literature  on  a  variety  of 
topics  related  to  military  staff  and  team  performance.  The  literatme  reviewed  includes  results  of 
earlier  MMBL  experimentation,  both  because  the  findings  were  important  in  shaping  the  general 
training  design,  and  because  the  MMBL  setting  was  selected  for  the  development  and 
implementation  of  the  prototype  training  package  described  later  in  this  report. 

There  were  three  major  aspects  of  training  that  were  examined  within  the  literature:  what 
to  train,  how  to  train,  and  how  to  provide  feedback.  First,  because  the  focus  was  on  staff  training 
in  a  digital  environment,  the  literature  on  various  aspects  of  team  performance  in  information- 
rich  environments  was  examined.  Second,  the  structure  of  the  training,  in  terms  of  the  order  of 
presentation,  pace,  and  delivery  mode,  was  examined  in  relation  to  the  content  areas.  Finally,  the 
literature  on  various  methods  for  obtaining  data,  both  to  assess  training  quality  and  to  provide 
feedback,  was  explored. 

Training  Methods  and  Models 

In  reviewing  the  literature  on  team  training,  four  common  content  themes  were  apparent: 
decision-making  models,  team  skills  (particularly  shared  mental  models),  information 
processing,  and  situational  awareness.  Because  of  the  changes  in  staff  structures  made  possible 
by  the  more  advanced  equipment,  decision-making  procedures  are  likely  to  be  modified,  and 
familiar  mental  models  shared  by  staffs  will  need  to  be  reexamined.  The  advanced  digital 
environments  will  provide  more  information,  more  accurately  and  more  quickly,  which  will 
accelerate  decision-making  while  stressing  information-processing  protocols.  The  potential  for 
improved  situational  awareness  will  only  be  realized  if  staffs  can  recognize  and  process 
information  relevant  to  the  situation  at  hand.  Thus,  all  of  these  content  areas  were  deemed  to 
have  particular  relevance  to  the  challenges  and  opportunities  within  the  reengineered  staff  and 
advanced  C^I  environments. 
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Decision-making  processes.  Army  training  in  the  staff  decision-making  process  has  not 
fully  utilized  the  potential  that  digitization  can  bring  to  battle  command,  especially  at  the  brigade 
and  battalion  tactical  levels  (Campbell,  Deter,  et  al.,  1999).  The  current  thrust  of  Army 
digitization  has  been  to  speed  up  the  decision-making  process  rather  than  improving  the  quality 
of  decision-making.  The  following  discussion  reviews  traditional  models  of  decision-making  as 
well  as  more  recent  theories  of  the  process.  Each  body  of  theory  is  also  associated  with  specific 
training  techniques  for  improving  decision-making  skills. 

Researchers  studying  traditional  decision-making  have  attempted  to  identify  a  generic 
decision-making  process  that  can  be  used  successfully  in  a  variety  of  contexts.  These  training 
efforts  have  been  directed  toward  how  best  to  teach  a  set  of  generic  decision-making  strategies. 
The  theoretical  model  underlying  the  traditional  decision-making  process  assumes  the  decision 
maker  is  a  rational/economic  human  (Simon,  1956)  or  vigilant  decision  maker  (Jams  &  Mann, 
1977)  who  systematically  searches  for  relevant  information  in  an  unbiased  manner  and  then 
carefully  weighs  the  utility  of  each  alternative  before  making  a  choice.  Early  training  efforte 
focused  on  how  to  train  people  to  use  decision  analytic  techniques  to  select  the  best  alternative 
course  of  action.  Increasingly,  however,  the  research  demonstrated  that  human  decision-makers 
often  fail  to  follow  the  prescriptions  of  normative  models. 

As  a  result,  training  efforts  became  focused  on  how  to  unbias  decision  makers.  Baron 
and  Brown  (1991)  provide  an  excellent  review  of  the  traditional  decision  skills  training  programs 
that  have  been  implemented  with  students.  Although  their  book  focuses  on  how  to  improve 
adolescents’  decision-making  ability,  their  model,  which  they  call  Personalized  Decision 
Analysis  (PDA),  is  also  used  by  adults  as  documented  in  a  variety  of  settings.  The  PDA  model 
assumes  the  decision-maker  is  faced  with  a  choice  between  options  (i.e.,  courses  of  action)  and 
needs  to  estimate  the  uncertainty  of  the  possible  outcomes  associated  with  each  option. 
Decision-makers  are  also  required  to  judge  the  desirability  of  each  outcome  in  terms  of  the 
expected  utility  to  be  gained  or  lost  should  this  outcome  occur.  The  mathematical  formulas  of 
statistical  decision  theory  (Savage,  1954)  are  then  used  to  determine  the  option  with  the  highest 
value  based  on  the  probability-weighted  utility  of  its  outcomes  (Raiffa,  1968). 

One  of  the  most  extensive  decision  training  programs  developed  from  the  decision 
analytic  perspective  is  the  Goals,  Options,  Facts,  Effects,  and  Review  (GOFER)  course 
developed  by  Maim  and  his  colleagues  (Mann,  Beswick,  Alloache,  &  Ivey,  1989;  Mann, 
Harmoni,  &  Power,  1991;  Mann,  Harmoni,  Power,  Beswick  &  Ormond,  1988).  The  GOFER 
course  is  based  on  Janis  and  Mann’s  (1977)  conflict  theory  of  decision  making  which  claims  that 
the  ideal  decision-making  process  is  that  of  the  “vigilant”  decision-maker.  Mann  et  al.  (1991) 
explain  that  the  acronym  GOFER  is  formed  from  the  five  criteria  for  vigilant  decision-making; 

•  goals  (surveying  values  and  objectives), 

•  options  (considering  a  wide  range  of  alternative  actions), 

•  facts  (searching  for  information), 

•  effects  (weighing  the  positive  and  negative  consequences  of  the  options),  and 

•  review  (planning  how  to  implement  the  options). 
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Mann  and  his  colleagues  note  that  additional  evaluation  studies  are  needed  to  determine  whether 
or  not  the  decision  skills  taught  in  the  GOFER  course  will  transfer  to  real-life  settings. 

There  has  been  limited  success  in  establishing  the  effectiveness  of  decision-skills  training 
programs  based  on  studies  of  traditional  decision-making  (Fallesen  &  Pounds,  1998).  Although 
several  controlled  experimental  studies  have  been  able  to  demonstrate  changes  in  decision¬ 
making  behavior  in  a  laboratory  or  classroom  context,  it  is  unclear  whether  or  not  these  results 
will  generalize  to  a  real-world  environment.  Recently,  decision  researchers  have  questioned 
whether  it  is  appropriate  to  attempt  to  unbias  decision-makers  when  the  biases  may  only  exist  in 
artificial,  laboratory  settings.  For  example,  Gigerenzer  (1991)  demonstrated  that  the 
overconfidence  bias  that  has  been  frequently  documented  in  the  laboratory  may  disappear  when 
the  decision  tasks  are  presented  in  more  meaningful  ways.  If  decision-makers  in  real-world 
settings  do  not  suffer  from  the  overconfidence  bias,  then  training  that  attempts  to  remedy  this 
bias  is  irrelevant. 

The  Army’s  Military  Decision-Making  Process  (MDMP)  has  many  of  the  characteristics 
of  earlier  decision-making  theories  discussed  by  Simon  (1956)  and  Janis  and  Mann  (1977).  As 
described  in  Field  Manual  (FM)  101-5  Staff  Organization  and  Operations  (Department  of  the 
Army  [DA],  1997a),  the  MDMP  is  an  analytical  process  for  military  problem  solving.  It 
involves  problem  definition,  delineation  of  alternative  scenarios  and  ways  to  counter  each 
alternative,  and  weighing  of  the  pros  and  cons  to  ascertain  the  most  advantageous  decision.  The 
MDMP  is  a  viable  model  for  decision-making  under  specific  (and  usually  rare)  conditions:  when 
options  are  fairly  clear-cut,  and  when  there  is  sufficient  time  to  wargame  alternative  decisions. 
While  learning  the  MDMP  is  not  difficult,  mastering  and  using  it  effectively  are  problematic. 
More  often  than  not,  the  pressures  of  mission,  enemy,  terrain,  friendly  troops,  time,  and  civilian 
considerations  (METT-TC)  force  shortcuts  or  unintended  alterations.  The  unreliability  and 
incompleteness  of  available  information  about  the  current  situation  and  changing  conditions  have 
always  induced  uncertainty  into  the  military  decision-making  process. 

With  the  advent  of  digital  technologies,  uncertainty  should,  theoretically,  be  reduced. 

The  MMBL’s  BCR  I  was  designed  to  evaluate  ways  to  help  the  commander  achieve  his  need  to 
visualize  the  battlefield  and  to  examine  the  training  that  would  be  required  for  future  operations 
(Elliott,  Sterling,  &  Lickteig,  1998;  MMBL,  1998a).  One  important  finding  from  this  BCR  I  was 
that  several  steps  of  the  MDMP  were  performed  in  a  faster,  more  parallel,  and  less  formal  style 
and  that  this  was  somewhat  more  effective  and  efficient  than  the  current  MDMP.  In  the  second 
BCR,  the  MDMP  was  again  noticeably  altered  and  resulted  in  an  even  more  streamlined  process 
compared  to  the  first  BCR  (EUiott,  1998). 

Research  began  to  emerge  in  the  1980s  that  focused  on  how  experts  made  decisions  in 
their  natural  environments  or  in  simulations  that  preserved  key  aspects  of  their  work 
environments.  The  general  decision-making  model,  referred  to  as  naturalistic  decision-making 
(NDM),  provides  a  basis  for  explaining  decision-naaking  processes  in  environments  where 
awareness  of  the  current  situation  changes  very  quickly.  In  contrast  to  traditional  research  in 
laboratory  settings,  NDM  researchers  discovered  that  experienced  decision-makers  spend  more 
time  and  effort  sizing  up  the  situation,  than  comparing  alternative  options  or  solutions.  Zsambok 
(1997)  defines  NDM  as  follows: 
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The  study  of  NDM  asks  how  experienced  people,  working  as  individuals  or 
groups  in  dynamic,  uncertain,  and  often  fast-paced  environments,  identify  and 
assess  their  situation,  make  decisions  and  take  actions  whose  consequences  are 
meaningful  to  them  and  to  the  larger  organization  in  which  they  operate,  (p.  5) 

Cohen,  Freeman,  and  their  colleagues  (Cohen,  Freeman,  &  Thompson,  in  preparation; 
Cohen,  Freeman,  &  Wolf,  1996)  have  developed  critical  thinking  skills  training  based  on  the 
NDM  perspective.  This  training  focuses  on  tactical  decisions  made  by  naval  officers  in  a  ship’s 
Combat  Information  Center  (CIC).  Its  development  is  based  on  interviews  conducted  with  naval 
officers  who  described  their  CIC  experiences  in  actual  combat  situations  such  as  the  Gulf  War 
(Kaempf,  Klein,  Thordsen,  &  Wolf,  1996). 

Cohen  et  al.  (in  preparation)  acknowledge  that  for  some  situations  in  the  Combat 
Information  Center  (CIC)  there  is  no  time  to  deliberate  over  the  most  appropriate  action  (e.g.,  in 
situations  that  involve  immediate  threat  to  own  ship).  In  such  situations,  a  recognition-based 
decision  model,  such  as  Klein’s  Recognition  Primed  Decision  (RPD)  model  (1997),  provides  a 
good  description  of  the  expert  decision-maker’s  thinking.  However,  Cohen  et  al.  argue  that  in 
other  situations  there  is  adequate  time  to  deliberate  about  the  decision  problem,  and  it  is  on  these 
situations  that  their  training  focuses.  An  important  part  of  Cohen  et  al.’s  critical  thinking 
training  involves  the  critiquing  process.  Decision-makers  are  taught  how  to  use  a  devil’s 
advocate  technique  to  uncover  hidden  assumptions  in  their  stories.  Decision-makers  are  told  that 
an  infallible  crystal  ball  indicates  that  their  current  assessments  are  wrong  and  encourages  them 
to  explain  how  that  could  be  the  case.  This  technique  alerts  the  decision-maker  to  consider 
significant  alternative  explanations  for  the  current  evidence. 

Cohen  et  al.  (in  preparation)  describe  their  efforts  to  evaluate  the  effectiveness  of  their 
critical  thinking  training.  A  total  of  95  military  officers  (a  mix  of  Navy,  Marine,  Army,  and  Air 
Force)  participated  in  two  separate  studies.  They  assessed  the  effectiveness  of  the  training  by 
having  participants  complete  questionnaires  about  a  scenario  similar  to  scenarios  used  during 
training.  The  participants  played  the  role  of  the  Tactical  Action  Officer  in  the  CIC.  Support  for 
the  effectiveness  of  Ae  critic^  thinking  skills  training  was  reflected  in  comparisons  between 
trained  and  untrained  officers  on  measures  such  as  the  number  of  conflicting  items  generated, 
and  the  number  of  alternative  explanations  generated.  In  addition,  the  accuracy  of  the 
participants’  assessments  was  evaluated  by  comparison  to  the  assessments  of  a  subject  matter 
expert  (SME).  In  general,  across  these  different  measures,  the  critical  thinking  skills  training 
was  demonstrated  to  increase  the  quality  of  the  officers’  decision-making  skills. 

One  of  the  more  comprehensive  cognitive  skills  training  programs  developed  to  date  in 
an  Army  context  was  developed  by  Fallesen  and  his  colleagues  at  ARI  (Fallesen,  1995;  Fallesen, 
Michel,  Lussier,  &  Pounds,  1996).  They  call  their  training  “Practical  Thinking”  to  contrast  it 
with  theoretical  or  formal  methods.  Practical  Thinking  training  focuses  on  cognitive  processes 
directed  toward  a  goal  and  performed  in  specific  circumstances.  The  Practical  Thinking  training 
developed  by  Fallesen  and  his  colleagues  includes  aspects  of  both  critical  and  creative  thinking; 
participants  are  trained  to  think  analytically  and  critically,  but  are  also  encouraged  to  be 
innovative  and  “daring.”  The  researchers  developed  course  materials  which  included  lessons  on 
multiple  perspectives  (i.e.,  thinking  outside  the  box),  metacognitive  skills  that  allow  the 
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individual  to  guide  his  or  her  thinking  deliberately,  techniques  for  identifying  hidden 
assumptions,  practical  reasoning  techniques  (e.g.,  demonstrations  of  reasoning  fallacies),  and 
integrative  thinking  techniques  to  increase  participants’  understanding  of  the  relationships  among 
events  and  concepts. 

In  summarizing  the  lessons  learned  from  administration  of  the  Practical  Thinking 
training,  Fallesen  et  al.  (1996)  concluded  that  Practical  Thinking  training  was  well  received  by 
the  majority  of  the  participants.  However,  they  acknowledged  that  “Practical  Thinking  is  not  so 
much  taught  as  it  is  modeled  and  encouraged  by  instructors  and  self-learned  by  students”  (p.  89). 
Therefore,  they  recommended  that  future  cognitive  skills  training  should  augment  class  time 
with  increased  practical  exercise  assignments  to  be  completed  by  students  outside  of  the 
classroom.  Fallesen  et  al.  also  suggested  that  future  research  is  needed  to  develop  a 
comprehensive  technique  to  evaluate  the  effectiveness  of  cognitive  skills  training. 

The  latest  revision  to  the  Army’s  MDMP  incorporates  some  of  the  techniques  of  NDM 
into  its  discussion  of  how  commanders  can  accelerate  the  decision-making  process  in  a  time 
constrained  environment.  Four  techniques  that  assist  in  accelerating  the  process  include:  a) 
expanding  the  commander’s  direct  involvement  in  the  decision-making  process;  b)  having  him 
issue  more  directive  guidance  to  his  subordinates;  c)  limiting  the  number  of  courses  of  action  to 
be  developed  and  war-gamed;  and  d)  maximizing  parallel  planning  among  the  staff  and  with 
subordinate  units  (DA,  1997a).  These  techniques  parallel  the  previously  discussed  literature 
about  how  expert  decision-makers  operate. 

Recently,  there  have  been  increasing  efforts  to  apply  principles  of  NDM  to  the  training  of 
cognitive  skills  of  various  teams,  including  battle  command  staffs  (e.g.,  Klein,  Orasanu, 
Calderwood,  &  Zsambok,  1993;  Klein,  1998;  Flin,  Salas,  Strub,  &  Martin,  1997;  Zsambok  & 
Klein,  1997).  In  particular,  the  naturalistic  framework  seeks  to  train  decision  skills  in  context. 
Rather  than  trying  to  teach  a  generic  method,  researchers  (e.g.,  Klein,  Kaempf,  Wolf,  Thordsen, 
&  Miller,  1997;  Klein,  McCloskey,  Pliske,  &  Schmitt,  1997)  have  described  the  importance  of 
identifying  the  decision  requirements  of  a  task:  the  difficult  and  critical  decisions,  the  reasons 
for  their  difficulty,  and  the  cues  and  strategies  used  to  handle  them. 

A  model  and  training  process  referred  to  as  Advanced  Team  Decision-Making  has  been 
used  by  Klein  Associates  since  1992  (Zsambok,  Klein,  Kyne,  &  Klinger,  1992).  In  their 
observation  and  analysis  of  teams  in  a  variety  of  settings,  they  found  few  advanced  teams, 
defined  as  teams  that  had  reached  their  fell  potential.  And  the  teams  that  did  perform  well  were 
unable  to  describe  what  they  did  or  reliably  replicate  that  performance  in  other  instances.  The 
researchers  therefore  described  a  decision-making  model  with  several  critical  behaviors  that 
distinguish  high  performance  teams  from  less  productive  ones.  The  model  proposes  three  basic 
components  of  advanced  team  decision-making:  team  identity,  team  conceptual  level,  and  team 
self-monitoring.  Strong  team  identity  is  associated  with  a  shared  understanding  of  individual 
roles  and  functions;  active  participation  by  all  team  members;  compensating  behaviors  that 
ensure  that  team  goals  are  achieved;  and  avoiding  micro-management  throughout  the  team. 

Team  conceptual  level  refers  to  the  extent  to  which  the  team  thinks,  solves  problems,  makes 
decisions,  and  takes  actions  collectively  on  a  level  of  complexity  and  sophistication  that  matches 
the  demands  of  the  task.  Teams  with  high  conceptual  levels  are  able  to  envision  common  goals 
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and  plans;  focus  their  decision-making  within  an  appropriate  span  of  time  and  on  relevant 
concepts  and  information;  discover  and  fill  holes  in  the  information  and  assumptions  and 
recognize  and  handle  inconsistencies;  and  accommodate  divergent  views  and  arrive  at 
converging  plans.  Team  self-monitoring  refers  to  the  team’s  ability  to  observe  itself  for 
consistently  high  team  identity  and  conceptual  levels.  It  involves  both  the  ability  to  adjust  based 
on  circumstances  and  the  ability  to  manage  time  effectively.  The  model  and  training  program 
formed  the  basis  for  training  at  the  Industrial  College  of  the  Armed  Forces  using  highly 
structured  low-fidelity  simulations  as  well  as  National  Security  Strategy  Exercises. 

The  theoretical  foundation  for  applying  NDM  to  the  decision  skills  training  used  by 
Klein,  McCloskey,  et  al.  (1997)  draws  directly  from  the  research  literature  on  expert-novice 
differences  (Drejius,  1972;  Dreyfus  &  Dreyfus,  1986;  Ericsson,  1996;  Hoffman,  1992).  There 
are  a  variety  of  differences  that  separate  novice,  or  near-novice  performers  from  highly  skilled 
ones.  These  include  differences  in  the  quantity  and  organization  of  what  is  known,  differences  in 
how  information  is  sensed  and  responded  to,  and  differences  in  problem-solving  and  reasoning 
strategies  (Chi  &  Bjork,  1991;  Ericsson  &  Smith,  1991;  Ford  &  Kraiger,  1995;  Glaser,  1990; 
Klein  &  Hoffman,  1993).  Moreover,  the  special  knowledge  and  skills  that  expert  performers 
possess  are  not  general  ones,  but  are  specific  to  the  tasks  and  domain  in  which  they  are  learned 
and  used  (Feltovich,  Ford,  &  Hoffman  1997). 

In  trying  to  identify  the  strategies  skilled  performers  use  to  develop  their  expertise,  the 
work  of  Klein  and  his  associates  was  particularly  relevant.  Klein,  McCloskey,  et  al.  (1997) 
developed  Decision  Skills  Training  workshops  that  require  the  participants  to  practice  using 
these  strategies.  They  have  presented  these  workshops  to  urban  and  commercial  firefighters,  and 
officers  and  enlisted  personnel  in  the  U.  S.  Marine  Corps  (USMC).  Most  of  these  workshops 
have  been  conducted  for  the  USMC,  and  these  efforts  are  briefly  summarized  next. 

The  first  Decision  Skills  Training  program  was  developed  in  response  to  a  request  from 
the  USMC  to  train  squad  leaders  to  become  more  effective  battlefield  decision  makers  (Klein, 
McCloskey,  et  al.,  1997;  McCloskey,  Lake,  Pliske,  &  Klein,  1998).  The  USMC  wanted 
Decision  Skills  Training  for  their  squad  leaders  because  they  were  experimenting  with  new 
concepts  of  battlefield  operations;  ones  in  which  small  teams  operate  independently.  The  leaders 
of  these  teams  were  being  faced  with  more  decision-making  responsibility  than  ever  before.  In 
response,  the  USMC  needed  an  effective  program  to  enhance  their  squad  leaders’ 
decision-making  skills  and  provide  them  with  opportunities  to  practice  these  skills.  The  training 
that  was  developed  for  the  Marines  provided  the  squad  leaders  with  tools  that  support  the 
strategies  for  achieving  expertise.  Thus,  the  squad  leaders  could  use  the  techniques  they  learned 
to  get  the  most  out  of  their  experiences,  and  optimize  their  learning,  just  as  experts  do. 

To  support  this  need,  Klein,  McCloskey,  et  al.  (1997)  reviewed  recent  work  in  the  field  of 
expertise  (Chi,  Glaser,  &  Farr,  1988;  Ericsson  &  Chamess,  1994;  Gopher,  Weil,  &  Bareket, 

1994;  Groen  &  Patel,  1990;  Shanteau,  1988).  From  the  literature,  they  identified  a  set  of 
strategies  that  appear  to  be  effective  in  enhancing  skills.  These  include:  engaging  in  deliberate 
practice,  so  that  each  opportunity  for  practice  has  a  goal  and  evaluation  criteria;  compiling  an 
extensive  experience  bank;  obtaining  feedback  that  is  accurate,  diagnostic,  and  reasonably 
timely;  building  mental  models;  and  obtaining  coaching.  Researchers  then  developed  methods  to 
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help  the  squad  leaders  take  advantage  of  those  strategies.  Some  of  the  methods  identified 
included: 

1.  Decision  Requirements  Exercise.  In  this  exercise,  the  squad  leaders  defined  the 
challenging  decisions  they  were  faced  with  on  the  battlefield,  and  identified  why  those 
decisions  were  challenging,  the  types  of  errors  that  were  commonly  made,  and  the 
strategies  that  could  be  used  to  de^  with  the  decisions.  After  performing  this  exercise, 
the  squad  leaders  had  a  better  sense  of  the  judgments  and  decisions  facing  them,  why 
they  were  difficult,  and  where  people  can  go  wrong. 

2.  Tactical  Decision  Games  (TDGs).  A  series  of  fifteen  TDGs  were  developed  and  the 
immediate  superiors  of  the  squad  leaders  were  trained  to  conduct  TDG  seminar  sessions. 
The  TDGs  were  tailored  to  the  types  of  decision-making  scenarios  that  the  squad  leaders 
would  likely  encounter  on  the  battlefield.  They  focused  on  giving  the  squad  leaders 
training  on  the  cognitively  challenging  aspects  of  their  job.  The  TDGs  were  low-fidelity, 
paper-and-pencil  simulations  of  incidents  that  might  occur  in  the  field.  Each  TDG 
presented  a  dilemma,  with  high  levels  of  uncertainty.  Participants  worked  under  time 
pressure,  with  only  3-5  minutes  to  consider  how  they  would  react.  Participants  were  then 
called  on  to  verbally  issue  their  orders  in  front  of  the  rest  of  the  class.  The  seminar  leader 
questioned  the  rationale  behind  the  participants’  actions,  and  other  students  were  also 
encouraged  to  question  the  reasoning  behind  the  orders.  These  TDGs  were  intended  to 
provide  indirect  experiences,  and  to  allow  practice  in  rapid  decision  making.  This 
training  was  later  extended  to  show  students  how  to  develop  their  own  TDGs  based  on 
the  decision  requirements  that  they  had  identified  earlier.  The  TDGs  that  were  developed 
for  the  squad  leaders  were  typically  practiced  at  the  rate  of  one  to  two  per  week. 

3.  Uncertainty  Matrix.  This  exercise  was  developed  to  help  people  reflect  on  their 
uncertainty-management  practices.  They  considered  the  uncertainty  that  they  were 
encountering,  how  they  were  dealing  with  that  uncertainty,  how  that  uncertainty  was 
affecting  their  performance,  and  how  they  might  better  deal  with  it.  As  a  result  of  this 
exercise,  participants  realized  that  they  were  dealing  ineffectively  with  their  uncertainties. 

4.  PreMortem  Exercise.  The  idea  of  a  PreMortem  is  to  identify  key  vulnerabilities  in  a  plan 
prior  to  execution.  After  someone  has  developed  a  plan,  the  team  then  spends  a  few 
minutes  trying  to  determine  where  the  plan  is  most  likely  to  fall  apart.  The  intent  is  to 
uncover  critical  flaws  and  areas  of  concern  that  are  otherwise  ignored.  The  PreMortem 
achieves  this  by  having  team  members  assume  up  front  that  the  plan  has  failed.  The 
challenge  then  becomes  uncovering  different  causes  for  this  failure.  These  flaws  can  then 
be  dealt  with  up  front,  while  the  plan  can  still  be  modified. 

5.  Post  Mortem.  This  tool  was  developed  to  have  the  squad  leaders  reflect  on  what  went 
well  and  not  so  well  during  an  exercise  (both  a  TDG  and  an  actual  field  training 
exercise),  and  to  use  this  reflection  to  increase  the  learning  from  experience.  Typically, 
after  an  exercise  there  would  be  an  after  action  review,  covering  the  actions  taken  and 
other  actions  that  might  have  been  better.  The  Post  Mortem  checklist  complemented  this 
by  addressing  processes.  It  allowed  the  participants  to  explore  patterns  that  might  have 
been  seen  earlier,  and  assessments  that  were  mistaken. 
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To  measure  the  effectiveness  of  the  training  program,  Klein,  McCIoskey,  et  al.  (1997) 
relied  heavily  on  subjective  evaluations.  The  participating  squad  leaders  and  their  immediate 
superiors  indicated  that  the  training  was  useful,  that  performance  improvements  had  occurred 
that  were  attributable  to  the  training,  and  that  they  would  be  likely  to  continue  to  use  the 
techniques. 

Teamwork  skills.  Research  shows  that  members  of  effective  teams  have  accurate  mental 
models  of  team  processes  that  are  shared  with  other  team  members.  Cannon-Bowers, 
Taiuienbaum,  Salas,  and  Volpe  (1995)  define  shared  mental  models  as  representations  of  how 
members’  roles  interact,  the  relationships  among  team  roles,  the  information  sources  that  are 
important,  and  the  appropriate  communication  channels  and  patterns  of  information  flow.  They 
are  “...preexisting  knowledge  structures  developed  over  time  and  generalized  to  a  variety  of 
situations”  (Cannon-Bowers  et  al.,  pp.  350-351). 

The  role  of  shared  mental  models  in  team  performance  is  to  provide  a  set  of  organized 
expectations  for  performance  by  which  to  draw  accurate,  timely  predictions  (Cannon-Bowers, 
Salas,  &  Converse,  1992;  Salas  &  Cannon-Bowers,  1997).  Some  advantages  of  shared  mental 
models  are  that  they  enable  team  members  to  generate  expectations  about  their  teammates’  roles, 
the  task  demands,  and  the  equipment  used.  Teams  with  these  knowledge  structures  are  more 
likely  to  have  accurate  expectations  in  regard  to  the  team’s  needs  and  requirements.  By 
extension,  they  should  be  able  to  respond  to  or  even  anticipate  needs  and  requirements,  thus 
resulting  in  more  proficient  team  performance. 

Cannon-Bowers  et  al.  (1992)  suggest  that  team  effectiveness  is  a  function  of  whether  the 
expectations  generated  from  team  members’  mental  models  are  compatible  with  the  expectations 
generated  by  other  team  members.  In  order  to  ensure  compatibility,  team  members  should  share 
their  mental  models  that  describe  interactions  with  one  another,  but  not  detailed  models  of  their 
individual  functions.  However,  other  team  members  should  possess  familiarity  with  those 
functions.  Therefore,  models  that  define  an  individual’s  contribution  to  the  team’s  task  need  to 
be  common  among  team  members.  In  addition,  models  that  create  expectations  about  how 
events  will  probably  occur  and  how  the  team  will  possibly  respond  to  task  demands  should  also 
be  shared.  The  authors  summarize  that  the  theory  of  shared  mental  models  does  not  mean 
identical  mental  models,  but  rather  compatible  mental  models  that  lead  to  common  expectations 
for  the  task  and  team. 

Salas,  Cannon-Bowers,  and  Johnston  (1997),  Johnston,  Smith- Jentsch,  and  Cannon- 
Bowers  (1997),  and  Cannon-Bowers  et  al.  (1992)  discuss  ways  to  develop  shared  mental  models. 
Their  techniques  include: 

•  training  team  leaders  to  encourage  team  members  to  discuss  their  roles  and  expectations 
with  other  team  members; 

•  training  the  team  members  on  teamwork  skills  such  as  situational  awareness,  leadership, 
and  techniques  to  share  workload  among  the  team; 

•  providing  experience  for  teams  to  function  under  the  types  of  stressful  conditions  they 
will  encounter  by  cross-training  (team  members  practice  on  the  roles  and  tasks  of  others 
so  they  understand  the  responsibilities  of  other  team  members); 
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•  incorporating  feedback  mechanisms,  such  as  debriefing  sessions,  into  training  to  improve 
accuracy  of  shared  mental  models; 

•  allowing  team  members  to  self-correct  their  performance  by  diagnosing  their  own 
performance  deficiencies  as  part  of  a  structured  process  that  provides  developmental 
feedback  at  the  completion  of  a  training  session;  and 

•  showing  leaders  how  to  maintain  shared  situational  awareness. 

Recent  research  on  training  and  assessing  shared  mental  models  was  reported  by  Stout, 
Cannon-Bowers,  Salas,  and  Milanovich  (1999).  They  explored  the  relationship  between  team 
planning,  shared  mental  models,  and  coordinated  team  decision-making  and  performance. 

Results  indicated  that  effective  planning  (as  indicated  by  activities  such  as  goal  setting,  sharing 
information,  and  clarifying  roles  and  responsibilities)  increased  the  formulation  of  shared  mental 
models  and  resulted  in  improved  coordinated  team  performance.  While  the  research  had  not 
been  reported  before  the  training  design  and  development  work  reported  here,  it  provides 
consistent  evidence  of  the  role  of  shared  mental  models  in  teamwork. 

In  the  literature  reviewed,  the  consideration  of  roles  and  function  is  closely  linked  to 
development  of  shared  mental  models.  The  BCR  I  (Elliott  et  al.,  1998;  MMBL  1998a)  examined 
the  roles  and  functions  that  would  be  assumed  by  staff  members  in  future  operations.  One  of  the 
findings  was  that  participant  roles  and  responsibilities  for  the  new  multifunctional  positions 
should  be  identified  and  communicated  to  the  participants  in  order  for  training  on  fiinctions  and 
tasks  to  be  effectively  accomplished. 

Situational  awareness.  Developing  shared  mental  models  among  staff  members  should 
lead  to  more  accurate  situational  awareness.  Although  definitions  of  situational  awareness  vary, 
Endsley’s  (1988)  is  the  most  often  cited  (e.g.,  Fracker,  1988;  Pew  &  Mavor,  1998;  Randel,  Pugh, 
&  Reed,  1996).  Endsley  defines  situational  awareness  as  “...the  perception  of  the  elements  in 
the  environment  within  a  volume  of  time  and  space,  the  comprehension  of  their  meaning,  and  the 
projection  of  their  status  in  the  near  future”  (p.  97).  Pew  and  Mavor  emphasize  that  for  the 
military,  situational  awareness  is  the  spatial  awareness  of  static  positions  of  self,  friendly  forces, 
and  enemy  forces  without  regard  to  their  movements.  They  suggest  that  this  view  should  be 
expanded  to  incorporate  Endsley’s  concept  of  the  dynamic  nature  of  the  battlespace.  The  1997 
edition  of  PM  101-5-1,  Operational  Terms  and  Graphics,  addresses  this  concern  by  the  use  of 
the  term  “battlefield  visu^ization.”  Battlefield  visualization  is  defined  as  “the  process  whereby 
the  commander  develops  a  clear  understanding  of  his  current  state  with  relation  to  the  enemy  and 
environment,  envisions  a  desired  end  state,  and  then  subsequently  visualizes  the  sequence  of 
activity  that  will  move  his  force  from  its  current  state  to  the  end  state”  (DA,  1997b,  p.  1-18). 

Situational  awareness  is  usually  studied  in  occupations  where  individuals  are  required  to 
make  quick  decisions  such  as  pilots,  air  traffic  controllers,  fire  fighters,  and  military  commanders 
(Randel  et  al.,  1996).  According  to  Endsley  (1995),  some  of  the  important  factors  affecting 
situational  awareness  include  stress,  system  complexity,  and  operator  workload.  A  certain 
amount  of  stress  can  increase  performance  by  increasing  attention  to  important  aspects  of  the 
situation.  However,  too  much  stress  causes  people  to  narrow  their  field  of  attention  to  a  limited 
number  of  aspects,  and  to  sample  only  dominant  or  probable  sources  of  information.  System 
complexity  also  negatively  affects  situational  awareness,  as  does  operator  workload.  The  effect 


11 


of  these  factors  may  be  moderated  by  the  degree  to  which  the  operator  has  developed  an 
appropriate  internal  representation  of  the  system  and  how  well  that  representation  fits  with  that 
of  other  team  members. 

An  aspect  of  situational  awareness,  known  as  crew  awareness  (Pew  &  Mavor,  1998), 
refers  to  the  extent  to  which  staff  members  have  a  shared  mental  model  of  what  is  occurring  and 
knowledge  of  how  other  members  are  perceiving  the  same  situation.  Expectancies  are  important 
for  coordinating  team  activities,  and  for  enabling  all  team  members  to  notice  discrepancies  and 
anomalies  and  thereby  to  notify  the  team  that  the  shared  situational  awareness  may  be  inaccurate. 
Development  of  crew  awareness  allows  the  staff  to  function  in  unison  toward  a  common  goal.  In 
a  recent  study  examining  decision-making  in  a  USMC  regimental  command  post,  the  staff  had 
problems  maintaining  a  shared  situational  awareness  (Klein  et  al.,  1996).  For  example, 
situational  awareness  seemed  to  only  flow  in  an  upward  direction  to  the  commander,  and 
subordinates  rarely  had  an  awareness  of  the  whole  picture.  Therefore,  the  staff  may  have  been 
executing  courses  of  action  without  completely  understanding  the  puipose. 

Situational  awareness  stresses  the  importance  of  training  teams  to  make  better  decisions, 
work  better  as  a  team,  and  learn  like  experts.  Prince,  Chidester,  Bowers,  and  Cannon-Bowers 
(1992)  have  developed  several  team  training  programs  for  the  Navy  using  the  concepts  of  shared 
mental  models  and  situational  awareness.  The  use  of  situational  awareness  to  focus  attention  on 
critical  cues  is  important  for  team  members,  and  one  of  the  challenges  is  to  find  ways  to 
selectively  share  information  so  that  information  overload  is  avoided.  Expectancies  are 
important  for  coordinating  team  activities,  and  for  enabling  all  team  members  to  notice 
discrepancies  and  anomalies  and  thereby  to  notify  the  team  that  the  shared  situational  awareness 
may  be  inaccurate. 

Klein,  McCIoskey,  et  al.  (1997)  have  developed  techniques  to  help  teams  gain  experience 
with  developing  situational  awareness  and  shared  mental  models.  One  useful  technique  is 
referred  to  as  roles  and  functions  definitions.  This  technique  identifies  the  extent  to  which  team 
members  understand  the  task  responsibilities,  expertise,  and  roles  of  every  other  member  and  the 
extent  to  which  they  understand  the  resources  required  by  the  team  for  performing  its  functions 
(e.g.,  email,  information  sources).  The  inclusion  of  resources  specifically  in  this  dimension  was 
influenced  by  the  work  of  Fleishman  and  Zaccarro  (1993)  and  Helmreich  and  Foushee  (1993). 

Another  technique  developed  by  Klein,  McCIoskey,  et  al.  (1997)  provides  the  team  with 
insight  into  how  the  team  members  think.  Different  team  members  have  different  assessments  of 
a  situation.  Even  the  clearest  spoken  or  written  intents  cannot  completely  solve  the  problem 
because  members  of  a  team  have  unique  past  experiences,  are  operating  in  slightly  different 
environments,  and  react  to  situations  differently.  The  technique,  presented  as  a  situational 
awareness  calibration  exercise,  provides  insight  into  how  team  members  perceive  the  same 
environment  by  asking  team  members  to  answer  specific  questions  at  critical  points  during  an 
exercise  or  simulation.  The  type  of  questions  in  the  military  domain  could  include  issues 
regarding  expectations,  important  environmental  cues,  prediction  of  future  events,  quality  of 
communication  among  team  members,  and  indicators  that  a  plan  is  either  succeeding  or  failing. 
Sample  questions  include:  what  is  our  primary  goal  right  now;  and  what  will  be  happening  10 
minutes  from  now.  These  data  can  then  be  used  to  evaluate  how  well  the  team  is 
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communicating,  to  better  understand  how  team  members  perceive  their  roles  and  functions  in  the 
bigger  picture,  and  to  generate  strategies  for  a  coherent  overall  team  situational  awareness. 

Information  management.  Information  management  is  an  essential  skill,  related  to  shared 
situational  awareness,  that  is  a  critical  aspect  of  effective  team  performance  (Serfaty  &  Entin, 
1997).  It  ensures  that  the  commander  and  his  staff  have  good  situational  awareness  based  on  the 
available  information.  It  also  enables  the  team  to  handle  the  flood  of  information  caused  by 
increasing  information  technologies,  sensors,  and  communication  channels.  A  critical  aspect  of 
information  management  is  that  the  required  information  reaches  the  right  person  in  time  for  it  to 
be  utilized  to  its  maximum  potential  (DA,  1997a).  Individuals  have  to  know  who  needs  what 
information,  when  they  need  it,  and  for  how  long  it  is  still  relevant.  Thus,  members  of  the  team 
know  each  others’  roles  and  functions  and  which  information  supports  those  roles  and  functions. 
In  time  of  high  stress,  it  also  requires  individuals  to  appropriately  filter  the  information  so  that 
the  coimnander,  or  recipient  of  the  information,  is  not  ipundated.  The  use  of  situational 
awareness  to  focus  attention  on  critical  cues  is  important  for  team  members,  and  one  of  the 
challenges  is  to  find  ways  to  selectively  share  information  so  that  information  overload  is 
avoided. 

Two  ways  of  training  teams  in  information  management  skills  are  the  Team  Adaptation 
and  Coordination  Training  (TACT)  (Serfaty  &  Entin,  1997),  and  Staff  Training  in  Information 
Management  (STIM)  (Freeman,  Cohen,  Serfaty,  &  Thompson,  1997).  These  training  methods 
emphasize  the  adaptation  of  communication  and  coordination  strategies  by  team  members  under 
variable  conditions  of  stress.  They  provide  shared  situational  awareness  and  shared 
understanding  of  roles  and  functions  among  team  members.  They  also  provide  strategies  to 
adapt  information  “push  and  pull”  to  the  changing  nature  of  the  task  environment  (e.g.,  due  to 
increased  time  pressure  or  increased  workload).  Information  push  and  pull  refers  to  the  passing 
of  information  from  one  source  to  another  (push)  and  the  requesting  or  seeking  of  information 
from  different  sources  (pull).  Research  by  Entin,  Serfaty,  and  Deckert  (1994)  indicates  that  team 
effectiveness  is  enhanc^  when  one  or  more  team  members  provide  information  b^ore  they  are 
requested  to  do  so  (push).  Furthermore,  providing  information  in  advance  appears  to  be 
particularly  beneficial  in  situations  characterized  by  increased  workload. 

Team  members  are  given  alternate  strategies  to  deal  with  the  push  and  pull.  They  are 
also  trained  to  recognize  when  situational  demands  are  changing  and  therefore  require  an 
adaptive  information  management  strategy.  One  method  is  for  the  team  leader  to  provide  regular 
situation  updates  so  that  team  members  know  the  current  situation  and  what  they  are  trying  to 
accomplish.  Another  method  is  to  provide  either  cross  training  or  explicitly  state  each  member’s 
role,  primary  tasks,  and  key  information  requirements.  This  method  for  training  team 
information  management  has  been  demonstrated  successfully  for  Navy  Anti-Air  Warfare  teams 
in  the  CIC  (Serfaty  &  Entin,  1997). 

Training  Stmctures 

The  second  major  aspect  of  training  that  would  impact  the  design  and  development  was 
considerations  of  training  structure,  that  is,  how  to  train.  Based  on  their  research,  Klein  et  al. 
(1996)  feel  that  when  new  digital  systems  are  used  on  the  battlefield,  they  are  often  not  used  to 
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their  full  potential.  They  attribute  this  to  lack  of  proper  training  of  the  operators  on  the 
equipment.  Proper  training  is  defined  as  not  only  the  content  of  the  training  (training  the  right 
topics),  but  also  organization  and  presentation  of  the  training. 

Research  in  cognitive  psychology  indicates  that  experts  differ  from  novices  not  just  in  the 
amount  they  know,  but  in  the  way  their  knowledge  is  organized  (Means,  Salas,  Crandall,  & 
Jacobs,  1993).  Experts  group  information  and  use  their  domain  knowledge  base  to  organize  and 
structure  new  information;  they  even  perceive  displays  differently  compared  to  novices.  An 
extension  of  this  finding  is  that  people  should  be  taught  not  only  how  to  think  like  experts,  but 
also  how  to  learn  like  experts. 

This  structuring  of  knowledge  by  experts,  and  the  effects  on  perceiving  and  learning  new 
material,  has  been  demonstrated  in  several  diverse  fields,  such  as  chess  (Chase  &  Simon,  1973), 
physics  (Chi,  Feltovich,  &  Glaser,  1981),  statistics  (Schoenfeld  &  Herrmann,  1982),  computer 
programming  (Adelson,  1981),  and  baseball  (Chiesi,  Spilich,  &  Voss,  1979).  The  differences 
between  experts  and  novices  in  organizing  knowledge  has  implications  for  training,  including  the 
training  of  military  decision-making  in  operational  settings. 

After  reviewing  the  literature  on  training  structure,  four  aspects  of  effective  training  were 
identified  that  were  particularly  relevant  to  the  proposed  training  development:  use  of  advance 
organizing  techniques,  part-task  training,  deliberate  practice,  and  context  relevance.  The 
supporting  literature  is  summarized  briefly  below. 

Advance  organizers.  When  experts  leam,  they  organize  the  new  material  by  reference  to 
known  concepts  and  prior  knowledge.  Smith,  Ford,  and  Kozlowski  (1997)  refer  to  this  type  of 
knowledge  organization  as  an  advance  organizer.  Not  only  is  it  a  tool  that  experts  have 
developed  for  themselves,  it  is  a  tool  that  can  be  given  to  non-experts.  Before  actual  training 
even  begins,  an  outline  or  overview  of  the  new  material  can  serve  as  an  advance  organizer. 
Advance  organizers  provide  an  initial  organizing  structure  that  allows  trainees  to  organize  and 
retain  the  material  to  be  learned.  They  are  usually  based  on  relevant  concepts  that  are  already  in 
the  trainee’s  cognitive  structure  and  these  concepts  are  used  as  part  of  the  organizing  framework. 
Cannon-Bowers,  Rhodenizer,  Salas,  and  Bowers  (1998)  define  advance  organizers  as  “...a 
category  of  activities  such  as  outlines,  text,  aural  descriptions,  diagrams,  and  graphic  organizers 
that  provide  the  trainee  with  a  structure  for  the  information  that  will  be  provided  in  the  practice 
environment”  (p.  298). 

An  example  of  an  advance  organizer  is  provided  by  Cannon-Bowers  et  al.  (1998).  When 
training  employees  to  use  a  complex  software  package,  advance  organizers  could  be  used  to 
display  the  organization  of  material  hierarchically  (to  correspond  to  pull-down  menus)  so  the 
learners  could  follow  it  as  they  leam  the  software.  As  they  practice  using  the  program,  they  will 
have  prior  knowledge  that  the  functions  are  hierarchically  related.  This  should  help  the  learners 
develop  appropriate  knowledge  stractures  and  consequently  improve  future  performance. 

According  to  West,  Farmer,  and  Wolff  (1991),  the  concept  of  an  advance  organizer  is 
based  on  the  premise  that  two  of  the  most  important  variables  in  teaching  are  what  the  learner 
already  knows  and  how  that  information  is  organized.  Presented  prior  to  the  new  material,  the 
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advance  organizer  serves  as  a  bridge,  or  transition  statement,  providing  a  brief  outline  of  the  new 
material.  The  advance  organizer  helps  learners  recall  what  they  know  about  a  topic,  particularly 
their  cognitive  framework  for  the  information,  and  transfer  that  knowledge,  or  framework,  to 
new  topics.  Without  such  an  understanding  of  the  connections,  learning  might  be  limited  to  rote 
memorization  and  quickly  lost. 

Advance  organizers  seem  to  be  more  effective  when  the  learner  is  unfamiliar  with  the 
material,  is  inexperienced,  or  is  expected  to  transfer  the  information  to  new  situations  (Mayer, 
1981).  West  et  al.  (1991)  suggest  that  the  strength  of  using  the  advance  organizers  is  that  they 
facilitate  long-term  recall  and  transfer  of  general  concepts.  The  effects  increase  over  time,  and 
when  the  instruction  lasts  several  days  rather  than  a  few  hours,  the  retention  effects  are  stronger. 
In  a  study  conducted  by  Kraiger,  Salas,  and  Cannon-Bowers  (1995),  participants  were  provided 
with  an  advance  organizer  either  before  or  after  completion  of  a  training  session  on  a  tactical 
decision-making  simulation.  Results  showed  that  the  quality  of  the  participants’  knowledge 
structures  correlated  with  performance  after  training  for  those  who  received  the  advance 
organizer  information  prior  to  training,  but  not  for  the  others. 

The  organization  of  the  advance  organizer  is,  of  course,  critical.  Cannon-Bowers  et  al. 
(1992)  suggest  that  presenting  learners  with  an  advance  organizer  and  presenting  training  in  a 
hierarchical  organization  will  lead  learners  to  acquire  the  intended  organizational  structure  of  the 
material.  Elliott  et  al.  (1998)  recommended  that  the  BCR  training  include  presentation  of  a 
flowchart  of  responsibilities  for  nodes  and  among  nodes,  as  a  way  of  organizing  and  clarifying 
the  information  being  presented.  Other  recommendations  were  to  brief  the  purpose  of  the  BCR 
and  its  goals  to  all  participants,  to  include  a  general  overview,  and  to  describe  primary  and 
secondary  responsibilities  for  each  position.  All  of  these  were  suggested  as  ways  of  assisting  the 
training  audience  to  more  quickly  grasp  what  they  were  to  do. 

Part-task  training.  Another  aspect  of  training  that  impacts  performance  is  whether  the 
task  is  trained  in  parts  or  as  a  complete  whole.  In  part-task  training,  parts  of  the  task  are 
presented  separately,  and  the  learner  practices  each  subtask  in  isolation  before  receiving  the 
whole  task.  Mane,  Adams,  and  Donchin  (1989)  examined  participants’  performance  on  a 
computer  game  after  receiving  part-  or  whole-task  training.  The  participants  who  received  part- 
task  training  performed  better  and  required  less  training  time  than  the  others.  Those  who 
received  whole-task  training  required  twice  the  amount  of  training  time. 

On  the  other  hand.  Stammers  (1982)  points  out  that  in  a  complex  task,  there  is  a  danger  in 
breaking  the  task  up  artificially.  The  danger  is  that  the  learner  may  learn  to  perform  the  subtask 
one  way  in  training,  only  to  find  that  the  nature  and  demands  of  performance  change  entirely 
when  the  rest  of  the  subtasks  are  added  and  performed  as  a  whole.  Therefore,  care  should  be 
taken  to  develop  training  that  avoids  this  potential  problem  by  not  teaching  everything  in 
isolation,  but  rather  adding  context  and  requiring  some  parallel  performance. 

A  similar  concept  is  what  Means  et  al.  (1993)  refer  to  as  scaffolding,  which  reduces 
cognitive  workload  in  the  early  stages  of  training.  In  this  method,  the  instructor  or  training 
system  initially  handles  some  portions  of  the  task  so  that  the  trainee  can  focus  on  the  more 
critical  aspects.  As  the  trainee  acquires  competence,  the  workload  gradually  increases  to  realistic 
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levels.  For  example,  early  training  might  not  require  the  trainee  to  consider  logistics,  but  would 
require  it  in  later  training. 

For  both  the  part-task  training  and  the  scaffolding  variant,  the  implication  is  that  simple 
parts  of  tasks  would  be  taught  first,  in  simple  sets.  These  would  be  gradually  brought  together 
into  more  complete  sets,  and  more  complex  elements  would  be  added,  until  the  learner  is 
practicing  the  whole  task  as  an  integrated  activity.  In  previous  BCRs  (Elliott  et  al.,  1998;  Elliott, 
1998),  training  was  conducted  by  teaching  each  function  of  the  digital  system  in  isolation. 
Learners  were  taught  the  purpose  of  the  buttons  in  the  order  they  appeared  on  the  screen,  rather 
than  in  a  hierarchical  structure.  This  was  an  extreme  form  of  part-task  training,  and  artificially 
segmented  the  task.  It  did  not  provide  learners  with  a  basic  structure  on  which  they  could  build 
their  training.  Elliott  et  al.  made  several  recommendations  on  the  basis  of  the  BCR  results, 
including  training  beyond  “switchology”  to  include  how  to  use  the  tools,  information  gained  by 
using  the  tools,  and  their  effect  on  visualizing  the  battlefield. 

Deliberate  practice.  In  addition  to  structuring  training  from  simple  parts  to  more  complex 
whole  tasks,  deliberate  practice  is  also  very  important.  Deliberate  practice  refers  to  more  than 
just  practice.  It  incorporates  the  concept  of  part-task  training  described  above,  in  that  the  task  is 
broken  down  into  smdler  tasks  increasing  in  difficulty.  In  addition  to  simply  practicing  the  task, 
the  learner  receives  guidance  and  feedback,  and  repeats  the  task  until  the  criterion  is  achieved. 
Frederiksen  and  White  (1989)  point  out  that  unguided  practice  is  no  more  effective  than  no 
practice  at  all. 

A  related  concept  is  structured  training  (Campbell,  Campbell,  Sanders,  Flynn,  &  Myers, 
1995).  Stmctured  training  combines  the  identification  of  training  objectives  with  deliberate 
construction  of  scenarios  or  simulations  to  cue  performance  of  the  objective  tasks.  For  entry 
level  training,  the  scenarios  may  be  very  highly  structured,  so  that  all  users  encounter  the  same 
sequence  of  events  at  the  same  level  of  difficulty.  For  more  proficient  trainees,  scenarios  and 
training  guidelines  allow  for  more  flexibility  in  how  the  events  are  presented,  how  quickly  things 
happen,  how  powerful  the  enemy  is,  and  so  on.  In  either  case,  structured  training  provides  for  a 
focus  on  critical  tasks  in  a  planned  sequence  of  performance  that  reinforces  learning  and  builds 
on  prior  experience.  The  value  of  the  training  is  enhanced  when  observers  are  on  hand  to  mentor 
and  coach  the  participants,  and  when  the  observers  are  aided  by  guides  to  the  critical 
performance  elements. 

Practice,  whether  deliberate  or  not,  is  critical.  Ericsson,  Krampe,  and  Tesch-Romer 
(1993)  discuss  the  importance  of  practice  in  gaining  expertise.  According  to  the  authors,  practice 
is  the  single  most  important  determinant  of  performance:  the  more  practice  the  better,  since 
skills  grow  best  incrementally.  Solid  learning  takes  time  and  requires  a  steady  progression  of 
slowly  increasing  effort  over  the  course  of  considerable  deliberate  practice.  In  an  experiment 
conducted  by  Jones  (1989),  participants  practiced  video  games  over  a  period  of  15  days  and  were 
tested  4  to  18  months  later.  Based  on  his  findings,  Jones  concluded  thk  when  participants  are 
given  a  generous  amount  of  time  for  practice  and  do,  in  fact,  practice  during  that  time,  all 
participants  learn  and  retain  their  skills  better  than  those  with  less  practice  time. 
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Means  et  al.  (1993)  describe  several  aspects  of  training  that  can  produce  the  expert-level 
decision-making  skills  that  combat  commanders  need.  One  of  those  ideas  is  training  to  achieve 
automatic  rather  than  controlled  processes.  Controlled  processing  is  slow  and  requires  cognitive 
resources  whereas  automatic  processing  is  rapid  and  can  continue  without  conscious  attention; 
thus,  it  can  be  accomplished  in  parallel  with  other  activities  without  degrading  performance. 

Such  training  requires  very  large  amounts  of  practice.  In  the  context  of  decision-making,  the 
goal  is  not  to  train  the  decision-making,  per  se,  but  to  train  the  other  task  components  to 
automaticity.  The  commander  can  then  make  decisions  without  having  to  concentrate  on  all  the 
underlying  skills.  In  the  research  described  above,  Jones  (1989)  found  that  individuals  who  learn 
rapidly  will  overpractice,  and  this  leads  to  the  tasks  being  performed  automatically.  This  is  an 
important  advantage  for  decision-making  in  combat  since  combat  decisions  involve  complex 
tasks,  multiple  goals,  and  uncertain  environments,  all  of  which  produce  a  heavy  workload. 

During  the  second  BCR,  participants  pointed  out  that  they  felt  the  need  for  structured 
training  with  more  practical  exercises  requiring  the  use  of  all  the  digital  tools  and  more  hands-on 
training  with  the  tools  (Elliott,  1998).  These  suggestions  tie  closely  to  the  notion  of  part-task 
training  as  well  as  deliberate  practice. 

Context-based  training.  Finally,  although  all  of  the  previously  mentioned  techniques  are 
important  for  performance  improvement,  their  influence  will  be  even  stronger  if  training  is 
provided  in  a  job-related  context.  According  to  Means  et  al.  (1993),  training  should  be 
conducted  .  .within  a  meaningful  (but  not  necessarily  whole)  task  context. . (p.  316). 

Salisbury  (1990)  suggests  that  people  are  better  at  remembering  meaningful  information; 
therefore,  the  goal  of  training  should  not  be  to  make  learners  memorize  information.  Rather, 
training  should  attempt  to  modify  tasks  which  do  not  have  much  intrinsic  meaning  into  more 
meaningful  concepts.  Finally,  Cannon-Bowers  and  Bell  (1997)  propose  that  embedded  training 
(i.e.,  training  in  the  operating  environment  so  that  training  takes  place  on  the  job)  is  one  of  the 
necessary  types  of  training  in  order  to  produce  effective  decision-makers. 

Cannon-Bowers  et  al.  (1992)  suggest  that  allowing  learners  to  practice  interacting  with  a 
system  without  any  guidance  or  context  will  not  lead  to  the  development  of  an  acceptable 
understanding  of  the  material.  Specific  instructional  guidance  should  be  provided,  and  the 
training  should  be  specific  to  the  job  context,  to  ensure  accurate  understanding  among  team 
members. 

Training  and  Performance  Evaluation  Methods 

As  techniques  for  training  digital  staffs  evolve  from  training  programs  for  analog  staffs, 
there  is  impetus  to  exploit  the  inherent  capabilities  of  the  hardware  and  software  the  information- 
age  staff  is  using  to  measure  its  level  of  proficiency  and  to  provide  feedback  on  its  performance 
(Dwyer,  Fowlkes,  Oser,  Salas,  &  Lane,  1997).  The  feedback  issue  arises  out  of  the  performance 
training  literature  discussed  in  the  previous  section.  A  recurring  theme  in  the  documentation  of 
training  research  is  the  need  for  feedback.  Within  the  NDM  framework,  research  on  the  learning 
styles  and  knowledge  structures  of  experts  suggests  that  feedback,  which  helps  the  learner  to 
analyze  and  categorize  an  experience,  will  result  in  more  long-lasting  performance  improvement 
(Chi  et  al.,  1988;  Shanteau,  1988). 


17 


A  review  of  the  training  literature  identified  the  “...lack  of  team  performance  measures 
as  a  hindrance  to  the  development  of  effective  team  training  systems”  (Dwyer  et  al.,  1997, 
p.l39).  McIntyre  and  Salas  (1995),  in  reporting  their  research  on  measuring  and  managing  team 
performance,  identified  issues  not  only  in  determining  what  training  interventions  could  be 
developed  to  enhance  team  performance,  but  also  a  bigger  issue  of  determining  if  the  training 
interventions  worked.  Dwyer  et  al.  described  the  measurement  of  actual  team  performance  as 
important  but  noted  that  it  was  done  less  frequently  when  assessing  team  training.  They  also 
described  the  typical  evaluation  of  training  as  including  reactions  to  the  training,  teamworic 
attitudes,  and  knowledge  of  key  teamwork  concepts. 

Other  researchers  involved  in  measuring  team  performance  (e.g.,  Cannon-Bowers  & 

Salas,  1997)  note  that  both  processes  and  outcomes  must  be  addressed.  Process  measures 
address  the  activities,  responses,  and  behaviors  that  people  use  to  accomplish  tasks.  They 
include  planning,  decision-making,  information  processing,  and  synchronization.  The  processes 
need  to  be  linked  to  outcomes  such  as  execution  activities  and  combat  effects  for  interpretation. 
Outcome  measures  are  usually  not  diagnostic  by  themselves  since  they  do  not  show  the 
underlying  causes  of  performance,  or  how  to  improve  it.  Thus,  it  is  necessary  to  collect  both 
process  and  outcome  measures  and  find  a  way  to  relate  them  to  each  other.  Another 
consideration  in  designing  measures  relates  to  levels  of  measurement.  Cannon-Bowers  and  Salas 
point  to  the  use  of  multiple  levels  as  particularly  important  in  team  performance  measurement. 
For  example,  assessing  individual  communication  skill  is  important  for  feedback  purposes,  but 
communication  skills  should  also  be  assessed  at  the  team  level  because  of  the  implications  for 
staff  processes  and  products. 

In  an  attempt  to  overcome  deficiencies  they  perceived  in  the  traditional  method  of 
evaluating  team  performance  which  relied  on  SMEs’  numerical  ratings  of  performance,  Dwyer  et 
al.  (1997)  developed  the  Target  Accepted  Responses  to  Generated  Events  or  Tasks  (TARGETS). 
By  identifying  or  embedding  events  in  training  exercises  that  would  provide  opportunities  to 
observe  behaviors  of  interest,  the  number  of  observations  could  be  controlled,  observation 
instruments  specific  to  the  event  could  be  developed,  and  assessor  biases  could  be  reduced  by 
creating  checklists  that  reduce  the  rating  of  a  behavior  to  yes,  it  was  performed,  or  no,  it  was  not 
performed. 

When  TARGETS  was  implemented  in  a  training  environment  where  the  participants  were 
geographically  dispersed,  several  lessons  were  learned  which  may  apply  to  designing 
measurement  systems  for  assessing  a  digital  staffs  training  performance.  Assessors  need  to 
have  and  maintain  access  to  the  trainees’  communications  nets.  A  backup  system  to  provide 
feedback  on  an  event  that  an  assessor  may  have  missed  because  of  conununication  difficulties  or 
other  unforeseen  circumstances  is  essential.  Converting  assessments  from  multiple  sources  and 
formats  into  a  single  feedback  format  is  time  consuming  and  might  detract  from  the  requirement 
to  provide  timely  feedback  to  the  trainees.  Spontaneous  events  might  not  fit  into  the  checklists 
or  other  instruments  designed  to  record  behavior  which  might  mean  that  creative  or 
unanticipated  behaviors  may  go  unreported.  Another  lesson  reported  was  that  one  of  strengths  of 
TARGETS  (structuring  the  observation  checklists  into  answering  yes  or  no  questions  as  one 
means  of  reducing  assessor  bias)  does  not  reflect  the  degree  to  which  a  behavior  may  have  been 
performed  in  terms  of  accuracy,  completeness,  timeliness,  and  so  forth.  A  proposed  solution  for 
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this  shortcoming  was  to  incorporate  observer  qualitative  comments  into  the  checklists  for 
behaviors  that  had  room  for  improvement  even  though  they  had  been  performed  by  the  trainees 
(Dwyer  et  al.,  1997). 

The  TACT  training,  previously  mentioned,  was  designed  to  look  at  how  teams  adapt  their 
decision-making  strategies  and  how  team  training  interventions  and  structural  reconfigurations 
can  best  contribute  to  the  team’s  ability  to  maintain  superior  performance  under  a  wide  range  of 
stressful  operational  conditions  (Entin  et  al.,  1994).  Using  trained  observers,  the  researchers 
measured  six  dimensions  of  teamwork:  team  orientation,  communications  behavior,  monitoring 
behavior,  feedback  behavior,  back-up  behavior,  and  coordination  behavior.  At  the  conclusion  of 
a  test  of  the  TACT  team  training  procedure,  teams  that  were  rated  higher  in  teamwork  behaviors 
performed  better  after  their  team  training  than  control  teams  which  had  not  received  the  training. 
The  implication  of  this  research  is  that  measuring  teamwork  behaviors  may  help  quantify  team 
performance.  Another  dimension  of  teamwork  measured  by  the  research  team  was  the 
communication  data  generated  during  the  research.  The  researchers  measured  the  total  amount, 
direction,  type,  and  content  of  the  communication.  In  general,  Entin  et  al.  found  that  measures  of 
communications  could  predict  the  success  of  team  training  interventions,  but  that  this  measure 
required  considerable  analysis  to  obtain  results. 

Pioneering  research  in  the  early  1970’s  on  battalion  command  and  control  demonstrated 
there  was  a  measurable  relationship  between  staff  competence  and  combat  effectiveness 
(Crumley,  1989).  Cmmley  notes  that  in  measuring  staff  competence,  there  have  been  two 
dissimilar  approaches.  One  approach  is  that  if  the  unit  accomplished  the  mission  then  the  staff 
process  was  effective.  The  second  approach  is  that  due  to  a  multitude  of  factors  external  to  the 
staff,  such  as  opposing  force  (OPFOR)  activity,  inappropriate  application  of  doctrine,  or 
deficient  training,  the  quality  of  the  staff  process  is  a  better  measure  of  staff  effectiveness.  On  the 
other  hand,  Olmstead  (1992)  fayors  process  over  outcome  in  terms  of  defining  success.  “Where 
effectiveness  is  the  ultimate  outcome  (mission  accomplishment,  achievement  of  objectives, 
productivity,  etc.),  competence  is  the  capability  of  the  organization  to  perform  the  critical 
functions  (processes)  that  lead  to  achievement  of  effectiveness”  (Olmstead,  1992,  p.  X-5). 

Three  methods  of  performance  measurement  used  in  previous  training  research  are: 
trainer  observers  or  assessors  collecting  critical  information  on  both  processes  and  outcomes, 
surveys,  and  interviews  (e.g.,  Campbell,  Deter,  et  al.,1999;  Koger  et  al.,  1998).  Together,  these 
methods  can  yield  a  multifacet^  look  at  team  processes  during  training.  A  fourth  approach,  less 
widely  used,  involves  automated  measures. 

Automated  measures  use  the  digital  information  being  exchanged  among  various  C^I 
systems  and  nodes  to  provide  process,  product,  and  outcome  information.  For  example,  previous 
ARI  research  (e.g.,  Lickteig  and  Emery,  1994)  demonstrated  the  utility  of  developing  automated 
measures  that  defined  information  acquisition  and  communications.  The  measures  were  message 
content,  message  source,  time  message  received,  time  message  opened,  action  taken  (relay, 
delete,  no  action),  time  action  taken,  and  direction  message  relayed.  Thus,  for  example,  they 
were  able  to  analyze  message  processing  speed  by  using  message  opening  time,  read  time,  relay 
time,  and  delete  time. 
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The  use  of  automated  measures  was  furthered  during  the  Combat  Vehicle  Command  and 
Control  (CVCC)  Evaluation  conducted  by  ARI  at  Fort  Knox  (Lickteig  &  Collins,  1995).  The 
objective  of  the  effort  was  to  determine  the  effect  of  vertically  integrating  digital  command  and 
control  systems  in  an  armor  battalion.  To  assist  in  this  evaluation,  a  total  of  64  automated 
measures  were  developed.  These  measures  were  based  on  the  tactical  battlefield  operating 
systems  functions  of  maneuver,  fire  support,  command  and  control,  and  intelligence.  The 
analysis  of  the  data  produced  by  these  measures  formed  the  basis  for  the  evaluation.  Several 
limitations  were  noted  in  using  automated  measures  in  this  evaluation.  Among  them  were  the 
need  for  additional  automated  routines  to  detect  misorientation  and  boundary  violations,  more 
precise  indictors  of  target  acquisition,  and  identification  of  the  sources  for  all  OPFOR  kills.  In 
short,  automated  measurement  in  digital  systems  can  provide  a  wealth  of  detailed  information 
about  activities  performed  on  the  systems,  but  the  challenge  is  to  make  sense  of  the  details. 

Conclusions 


The  review  of  the  literature  on  training  content,  training  structures,  and  measurement 
methods  led  to  specific  recommendations  concerning  training  for  future  staffs.  The  training 
outline  (described  later  in  this  report)  incorporates  elements  of  an  advance  organizer  around 
which  to  structure  learning;  implements  part-task  training  presented  within  a  meaningful  context; 
and  provides  learners  the  opportunity  for  guided  practice  and  feedback.  By  using  these  concepts 
in  conjunction  with  structured  training  (for  a  review,  see  Campbell,  Deter,  &  Quinkert,  1997), 
staff  members  can  gain  a  better  understanding  of  why  they  are  doing  what  they  are  doing  as  well 
as  how  to  use  the  system  in  an  operational  context. 

On  the  content  of  training  for  staffs  in  digital  environments,  the  literature  indicates  that 
decision-making  skills  can  be  improved  with  training  that  focuses  on  pre-action  analysis, 
examination  of  uncertainties  and  weaknesses  in  a  plan,  and  post-action  discussion  and 
reinforcement  of  lessons  learned.  Team  orientation  can  be  enhanced  by  training  focused  on 
understanding  roles  and  functions,  maintaining  situational  awareness,  and  attending  to 
information  flow  management.  The  training  design  discussed  in  this  report  includes  elements 
designed  to  address  both  decision-making  skills  training  and  team  orientation. 

Review  of  the  literature  on  measurement  led  to  the  recommendation  of  a  multifaceted 
approach  to  address  two  measurement  needs:  assessing  training  quality  and  providing  staff 
performance  feedback.  The  measurement  approach  described  in  this  report  addresses  both 
individual  and  team  process  and  outcome  results. 

Prototype  Training  and  Evaluation  Methods 

The  design  for  the  training  of  future  staffs  is  described  here  in  terms  of  four  aspects,  all 
based  on  previous  research.  Three  of  those  aspects  were  developed  on  the  basis  of  the  literature 
reviewed  above:  training  content,  stracture,  and  assessment.  The  fourth  -  the  design  and 
development  methodology  -  is  derived  from  the  published  ARI  methodologies  for  development 
of  structured  training  described  previously.  Although  the  design  is  general  enough  to  be  used  for 
future  staff  training  under  a  range  of  conditions,  there  are  four  underlying  assumptions. 
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First,  staffs  are  already  moderately  proficient  in  their  individual  skills  and  familiar  with 
basic  staff  operations.  They  need  not  be  highly  experienced,  nor  is  it  necessary  that  they  have 
worked  together  as  a  staff  for  any  great  length  of  time. 

Second,  training  participants  are  not  yet  familiar  with  certain  key  elements  of  their 
environment.  Whether  they  are  training  up  for  a  new  type  of  staff  organization,  a  new  command 
post  setup,  or  new  digital  capabilities,  the  training  will  assist  them  to  both  learn  the  novel 
elements  and  also  become  more  proficient  as  a  staff. 

Third,  they  are  operating  in  an  information-rich  environment.  The  training  design  is 
specifically  aimed  at  operating  environments  wherein  staff  members  deal  with  complex 
information  management  and  communication  issues.  This  assumption  is  not  a  necessary  one;  the 
procedures  and  techniques  should  also  be  useful  in  conventional  (analog)  environments. 

Fourth,  some  type  of  virtual  or  constructive  simulation  will  be  used.  This  also  is  not  a 
necessary  assumption.  Although  the  use  of  simulation  greatly  aids  in  providing  realistic  contexts 
and  automated  measurements,  the  training  design  can  be  used  in  live  simulation  settings  as  well. 

Training  Content  for  Training  of  Future  Staffs 

The  content  of  the  training  for  future  staffs  should  cover  three  training  needs:  system 
operations,  tactical  skills  using  the  capabilities  of  the  new  environment,  and  team  process  skills 
to  bring  the  staff  quickly  to  a  high  level  of  proficiency.  The  system  operations  training  is 
primarily  individual  training.  Each  staff  member  needs  to  know  how  his  or  her  job 
responsibilities  are  performed  using  the  digital  equipment.  This  will  include  not  only  basic 
instruction  on  button-pushing  and  system  capabilities,  but  some  training  in  techniques  and 
procedures  for  using  the  system  efficiently. 

The  tactical  skills  training  is  more  advanced  than  system  operations.  It  requires  that  staff 
members,  after  becoming  individually  proficient  with  the  C*!  system,  work  in  small  groups  to 
use  their  new  capabilities  to  perform  small  parts  of  their  jobs.  The  training  would  gradually 
build  up  to  having  the  full  staff  performing  in  an  integrated  and  self-directed  way,  responding  to 
job  situational  cues  and  actually  performing  the  appropriate  actions. 

The  team  process  skills  training  should  include  specific  information,  practice,  and 
feedback  addressing  the  four  content  areas  described  earlier,  decision-making,  situational 
awareness,  teamwork  skills,  and  information  management.  These  need  not  be  treated  as  separate 
topics,  any  more  than  they  can  be  separated  during  performance.  They  should  also  not  be 
separated  from  the  job  situation,  as  generic  skills,  but  rather  should  be  practiced  and  discussed  as 
an  integral  part  of  the  mission  planning,  preparation,  and  execution  cycle. 

Training  Structure  for  Training  of  Future  Staffs 

The  recommended  structure  of  the  training  incorporates  aspects  described  earlier: 
advance  organizing  principles,  part-task  training,  deliberate  practice,  and  context-based  training. 
The  structure  described  here  and  eventually  used  in  the  prototype  training  package  is  very  similar 
to  that  of  the  structured  simulation-based  training  that  ARI  has  been  developing  since  1993 
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(Campbell,  Deter  et  al.,  1999;  Campbell,  Graves,  Deter,  &  Quinkert,  1998;  Campbell,  Pratt  et  al., 
1999;  Flynn,  Campbell,  Myers,  &  Burnside,  1998;  Hoffman,  Graves,  Koger,  Flynn,  &  Sever, 
1995). 


The  initial  information  presentations  to  the  participants  should  include  multiple  instances 
of  advance  organizers.  For  whatever  is  novel  within  the  foture  environment,  the  training 
audience  should  be  specifically  instructed  on  its  relation  to  the  “old”  way  of  doing  business,  and 
told  why  the  change  has  been  made.  If  there  are  elements  of  the  new  environment  that  are 
unchanged,  these  should  also  be  pointed  out  to  the  training  audience.  Job  aids  that  graphically 
show  the  old,  the  new,  and  the  linkage  should  be  used  where  possible. 

Incorporating  principles  of  part-task  training  means  that  the  system  training,  tactical  skills 
training,  and  team  process  skills  training  will  each  be  broken  down  into  smaller  chunks. 

Training  participants  will  receive  training  on  small  segments  of  tasks  and  may  even  be  clustered 
into  small  groups  to  receive  specific  system  training.  As  the  training  proceeds,  the  task  segments 
will  be  aggregated  and  the  small  groups  brought  together  to  allow  fully  integrated  practice  of 
tasks  and  activities. 

As  the  part-task  training  occurs,  there  should  be  numerous  planned  opportunities  for 
deliberate  practice  of  the  skills  just  taught.  This  practice  time  should  be  structured  to  focus  on 
the  skills  and  integration  of  previously  taught  skills,  and  should  focus  on  training  to  criterion. 
That  is,  not  only  should  ample  time  be  provided,  but  participants  should  demonstrate  their  grasp 
of  the  skills  several  times  (thus  minimizing  lucky  guesses  and  allowing  for  overpractice)  before 
moving  on  to  the  next  segment.  Feedback  is  a  necessary  component  of  deliberate  practice; 
participants  should  not  have  to  decide  for  themselves  whether  or  not  they  have  achieved 
proficiency. 

Finally,  all  of  the  training,  from  the  basic  and  advanced  system  training  to  fully  integrated 
full  staff  exercises,  should  be  presented  in  the  context  of  realistic  job  requirements.  This  can  be 
accomplished  even  for  the  simplest  task  segments  in  the  system  training,  by  telling  or  showing 
the  participants  how  the  skill  will  be  used  as  they  perform  their  jobs.  The  complex  integrated 
exercises  that  come  toward  the  end  of  training  can  initially  have  simple  scenarios,  with  later 
exercises  stressing  the  participants  with  very  high-intensity  scenarios  (e.g.,  less  time,  more 
enemy,  conflicting  information,  degraded  conditions). 

Training  and  Performance  Evaluation  of  Fumre  Staffs 

In  order  to  evaluate  training  and  performance  for  future  staffs,  a  combination  of 
automated  measures,  surveys,  observations,  and  interviews  should  be  used.  Specifications  for 
the  evaluation  plan  would  begin  with  an  analysis  of  the  specific  staff  task,  process,  or  product  to 
be  measured  and  a  determination  made  as  to  the  method  or  methods  that  could  best  be  used  to 
evaluate  each  task,  process,  or  product.  The  measures  would  then  be  designed  and  developed  to 
provide  converging  and  efficient  coverage  of  staff  performance  issues.  Devices  such  as  laptop 
computers  or  personal  data  assistants  should  be  incorporated  for  reducing  the  burden  of 
observation  and  survey  data  collection,  analysis,  and  formatting  so  that  immediate  training 
feedback  could  be  provided  to  the  staff. 
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Methodology  for  Design  and  Development  of  Future  Staff  Training 


The  development  of  structured  training  for  a  future  staff  should  follow  the  Army’s 
systems  approach  to  training  (DA,  1999)  and  the  procedures  outlined  in  Campbell  and  Deter 
(1997).  The  development  process  should  begin  with  a  thorough  front-end  analysis  of  what  is 
known  and  what  ne^s  to  be  learned  about  the  training  needs,  training  conditions,  and  training 
audience.  To  the  extent  possible,  all  training  requirements  and  resources  should  be  documented 
(or  at  least  noted  as  information  needs),  and  consensus  on  them  obtained  from  those  who  have 
responsibility  for  the  training  and  performance  of  the  training  audience. 

Analysis  then  leads  into  design  and  development,  resulting  in  products  that  should  be 
appropriate  for  the  conditions  imposed  by  the  environment  and  that  should  satisfy  the  training 
requirement.  The  design  and  development  work  incorporates  the  training  content  and  structure 
decisions  described  above,  and  includes  provisions  to  measurement  for  feedback  and  program 
evaluation. 

Throughout  the  analysis,  design,  and  development  activities,  training  products  should  be 
under  constant  review  and  revised  as  necessary.  During  initial  implementation,  formative 
evaluation  of  the  training  products  and  process  should  be  conducted  to  assess  success  in  meeting 
the  training  requirements.  Evaluation  should  continue  for  additional  implementations  of  the 
training,  so  that  developers  can  track  performance  deficiencies  that  should  be  included  in  the 
training  and  training  deficiencies  that  should  be  corrected. 

Training  Analysis 

As  stated  above,  training  design  and  development  relies  on  a  thorough  front-end  analysis 
(Campbell  &  Deter,  1997;  DA,  1999).  Such  analysis  includes  examination  of  the  training 
audience,  their  job  requirements,  and  a  complete  delineation  of  the  job  conditions  that  will  be 
represented  in  Ae  training.  As  a  starting  point  for  this  analysis,  the  project  team  used  the 
MMBL’s  future  battalion  battle  staff  concept  which  had  been  the  focus  of  recent  experimentation 
at  Fort  Knox.  This  future  battalion  battle  staff  is  led  by  a  conunander,  assisted  by  13  principal 
staff  officers  and  non-commissioned  officers,  which  are  deployed  in  four  command  and  control 
vehicles  (nodes)  and  operate  advanced  C*I  systems.  The  MMBL  has  used  an  enhanced  Modular 
Semi-Automated  Forces  (ModSAF)  system  to  emulate  advanced  C^I  systems.  These  systems 
are  referred  to  collectively  as  the  surrogate  command,  control,  communications,  and  computers 
(SC^)  system.  While  there  was  no  published  operational  or  procedural  doctrine  for  this  staff,  the 
project  team  was  given  access  to  previous  reports  on  the  MMBL’s  experimentation  which 
provided  a  starting  point. 

The  analysis  process  underlying  design  of  the  prototype  training  package  included; 

•  defining  the  battalion  staff  processes  in  terms  of  responsibilities  for  nodes  and  for 
individuals  within  nodes; 

•  defining  how  the  battalion  staff  processes  should  (or  could)  be  performed  within  the 
nodes; 

•  defining  what  the  SC^  tools  are  and  what  they  can  do;  and 

•  defining  individual  functions  and  tasks  using  the  SC^  tools. 


23 


These  processes  were  not  completely  sequential.  Some  were  parallel  activities,  and  every 
product  underwent  continual  modifications  to  ensure  that  it  remained  consistent  with  respect  to 
other  products.  If  there  had  been  no  previous  documentation  concerning  the  tools,  tasks,  and 
roles,  it  is  likely  that  the  process  would  have  proceeded  in  a  very  different  order.  However,  the 
discussion  of  the  analysis  process  below  is  presented  in  the  order  that  represents  how  the  work 
generally  proceeded  and  also  shows  most  clearly  the  developmental  and  conceptual  linkages. 
Products  for  each  of  these  analysis  components  are  contained  in  Training  and  Measurement 
Support  Package  Volume  1  (1999)  and  examples  of  the  materials  can  be  seen  in  Appendix  B. 

Descriptions  of  the  node  and  individual  job  responsibilities  were  derived  from  an 
integration  of  results  from  the  BCR  n  (Elliott,  1998),  subject  matter  expertise  on  Army 
operations  at  battalion  level,  and  doctrinal  materials  such  as  FM  101-5  (DA,  1997a).  The  subject 
matter  expertise  and  doctrinal  reference  were  used  to  define  the  staff  process  objectives,  which 
are  essentially  unchanged  from  current  requirements.  That  is,  the  staff  is  still  responsible  for 
planning,  maneuvering,  monitoring  enemy  activity,  supplying  the  force,  and  so  on.  However, 
the  reengineered  staff  environment  does  change  the  way  in  which  those  things  are  done  and  who 
is  responsible  for  doing  them. 

Examination  of  BCR  n  results  and  discussions  with  individuals  who  were  involved  in 
BCR  n  led  to  a  set  of  very  brief  descriptions  of  the  roles  and  responsibilities  of  each  of  the  four 
primary  nodes  and  of  the  individuals  within  the  nodes.  The  descriptions  were  also  reviewed  by 
MMBL  subject  matter  experts.  The  purpose  of  these  descriptions  was  twofold:  to  provide  the 
starting  point  for  more  detailed  performance  analyses,  and  to  provide  guidance  to  the 
participating  unit  as  they  started  training  on  how  their  nodes  and  positions  were  assumed  to 
function. 

In  order  to  develop  staff  training  objectives,  the  node  responsibility  and  individual  job 
responsibility  descriptions  were  expanded  using  documentation  from  BCR  11,  along  with 
reference  to  current  military  doctrine.  The  expanded  descriptions  took  the  form  of  performance 
analysis  worksheets  that  showed  what  each  individual  in  the  four  primary  nodes  would  need  to 
be  doing  during  the  plarming  and  execution  phases  of  a  mission.  The  performance  analysis  did 
not  attempt  to  account  for  all  of  the  staff  input  and  outputs  during  the  MDMP.  The  analysis  did 
account  for  the  capabilities  of  the  SC*  system  to  display  information  that  the  staff  would 
normally  have  to  collect,  analyze,  and  ^stribute,  and  concentrated  on  those  aspects  of  mission 
planning  that  would  be  different  for  a  future  staff,  geographically  dispersed  and  equipped  with  an 
advanced  C^I  system.  The  two  aspects  of  the  MDMP  that  were  judged  to  be  most  different  for 
this  staff  were:  course  of  action  development,  analysis  (to  include  wargaming),  comparison,  and 
selection;  and  rehearsals.  The  ability  to  issue  fragmentary  orders  in  response  to  rapidly  changing 
battlefield  conditions  and  to  higher  headquarters’  directed  changes  to  the  vuiit’s  mission  was  also 
thought  to  be  an  important  future  staff  attribute.  This  analysis  was  then  used  as  the  basis  for 
outlining  staff  training  for  four  segments  of  a  unit’s  mission  cycle  which  were  designated  as 
course  of  action  (COA)  development  (later,  mission  analysis),  rehearsal,  battalion  branch,  and 
brigade  sequel. 

The  performance  analysis  worksheets  initially  contained  only  the  events  within  the  four 
segments  with  indications  of  staff  activities  that  would  be  triggered  by  each  event  or  by  activity 
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within  the  staff.  After  the  analysis  of  the  SC^  tools  and  task  functions  (the  descriptions, 
crosswalks,  and  comparisons  described  in  the  following  paragraphs)  was  completed,  information 
on  which  tools  would  likely  be  used  in  which  activities  was  added.  The  performance  analysis 
worksheets  were  reviewed  by  the  MMBL  staff  before  being  used  for  exercise  construction.  The 
performance  analysis  information  was  also  used  to  derive  a  listing  of  more  general  functions  that 
underlie  task  and  activity  performance. 

In  order  to  provide  recommendations  concerning  the  SC'*  training,  a  complete  and 
accurate  listing  of  the  tools  and  their  use  was  needed.  The  SC^  tool  descriptions  and  their 
utilization  were  initially  drawn  from  the  Battle  Command  Reengineering  II  User’s  Manual  (HI 
(MMBL,  1998c).  The  descriptions  were  later  revised  as  the  system  underwent  upgrades.  The 
final  descriptions  were  reviewed  by  MMBL  staff,  and  were  also  verified  by  project  staff  working 
on  the  SC*  equipment.  Because  the  Battle  Command  Reengineering  III  User’s  Manual 
(Mounted  Maneuver  Battlespace  Lab,  1999a)  was  undergoing  modifications  in  parallel  with 
changes  to  the  SC^  system  itself,  the  use  of  the  manual  as  a  source  of  information  was  limited. 
Therefore,  project  staff  worked  closely  with  the  system  programmers  to  ensure  that  the  most 
current  configuration  and  operation  were  represented  in  the  training  design  work.  Once  the  SC^ 
tools  descriptions  were  completed,  they  were  used  to  determine  how  the  tools  should  be  used  to 
accomplish  the  general  task  functions  defined  in  the  earlier  performance  analysis. 

The  general  function  descriptions  allowed  the  determination  of  the  functional  training 
that  should  be  provided  for  the  training  audience.  The  function  descriptions  were  then 
crosswalked  to  tools,  and  also  crosswalked  to  individuals  within  the  four  primary  nodes.  All  of 
the  definitions  and  crosswalks  were  reviewed  by  MMBL  staff,  and  the  functions  descriptions  and 
crosswalk  to  tools  were  verified  on  the  SC^  equipment.  The  crosswalks  were  used  to  group  SC^ 
tools  for  purposes  of  initial  training  and  to  identify  logical  groupings  of  training  participants  who 
should  receive  training  on  particular  fimctions  and  tools.  Knally,  a  summary  of  the  SCT  tools  by 
training  level  was  created  to  ensure  all  tools  were  identified  and  would  receive  adequate  training. 

In  addition  to  the  analysis  of  performance  of  the  primary  training  audience,  the 
responsibilities  for  the  extended  training  audience  were  also  examined.  Close  examination  of 
BCR  n  operations  and  BCR  HI  plans  showed  the  responsibilities  and  tasks  to  be  the  same  as 
documented  in  current  doctrine. 

Training  Design 

The  analysis  served  as  the  foundation  for  design  of  the  training  that  focused  on  staff 
performance.  The  literature  indicated  that  staff  performance  could  be  improved  with  training 
that  incorporates  NDM,  provides  tools  to  increase  the  shared  understanding  of  the  battlefield, 
and  allows  for  the  practice  of  new  skills.  This  type  of  training  is  most  effective  when  presented 
in  a  scenario-based  context. 

The  staff  training  was  designed  to  focus  on  collective  decision-making  skills  and  team 
orientation  within  a  scenario  context.  Early  in  the  analysis  process  (previously  described), 
however,  it  became  evident  that  much  of  the  success  of  the  collective  training  would  be 
dependent  upon  the  level  of  proficiency  the  unit  members  would  have  with  the  SC^  system.  To 
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incoqjorate  aspects  such  as  part-task  training,  structured  training,  and  deliberate  practice,  design 
recommendations  for  the  initial  SC'*  systems  training  were  prepared.  These  recommendations 
covered  the  full  train-up  process  from  initial  orientation  through  individual  and  small  group 
training  to  participation  in  the  structured  collective  training  exercises. 

The  design  was  hierarchical,  outlining  a  5-level  training  sequence  (see  Figure  1).  It 
assumed  that  participants  would  have  some  experience  working  together  as  a  team.  The  training 
would  proceed  through  orientation,  individual  training,  small  group  training,  and  a  series  of 
tactical  decision-making  exercises  (TDXs).  The  purpose  of  each  level  is  described  below. 


Figure  1.  Training  level  sequence. 

In  Level  1,  the  training  participants  began  with  an  introduction  and  orientation  to  the 
train-up  and  the  trials.  The  Level  1  session  was  designed  to  serve  as  an  advance  organizer,  in 
accordance  with  research  indicating  that  learners  do  better  if  they  are  told  what  they  will  be 
learning,  how  they  will  receive  it,  and  how  it  relates  to  current  knowledge  and  processes.  In  this 
case,  the  unit  was  to  be  provided  with  general  information  regarding  the  training  facility,  the 
train-up,  the  data  collection,  and  the  activities  following  training.  This  orientation  was  designed 
to  be  delivered  either  as  part  of  the  advance  visit  to  the  unit  approximately  one  month  before  the 
training  was  to  start,  or  as  part  of  the  initial  session  when  the  unit  arrived  at  the  training  site.  The 
session  was  estimated  to  require  a  total  of  1-2  hours. 

The  fundamentals  training.  Level  2,  was  designed  to  provide  SC^  equipment  familiarity 
combined  with  general  system  operations.  At  this  stage  of  training,  the  information  was  basic 
and  focused  on  the  individual  vwth  a  limited  context  (e.g.,  windows  manipulation,  operating 
communications  equipment).  It  was  to  be  conducted  vnth  the  training  audience  in  their  primary 
seat  locations  within  the  nodes.  A  research  assistant  (RA)  was  assigned  to  each  node  and  was 
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responsible  for  delivering  the  training.  Level  2  training  was  estimated  at  about  2-3  hours,  so  that 
individuals  could  quickly  move  beyond  switchology  to  learning  job-related  functions. 

The  Level  3  training  was  then  designed  to  provide  the  context-based  training  by  grouping 
the  system  tools  into  functional  groups  (e.g.,  create  overlays,  conduct  a  whiteboard 
teleconference).  The  groupings  for  the  Level  3  training  recommendations  were  a  result  of  the 
analysis  of  the  SC^  tools  and  task  functions  analysis  discussed  in  the  previous  section.  Though 
this  training  still  focused  primarily  on  the  individual,  it  grouped  together  individuals  from 
different  nodes  who  performed  similar  tasks.  In  addition  to  learning  the  system,  this 
arrangement  provided  unit  members  the  opportunity  to  discuss  how  they  planned  to  use  the 
system  in  the  context  of  their  jobs  in  the  different  nodes.  Level  3  training  was  expected  to  be 
lengthy,  requiring  4-6  hours.  In  order  to  provide  deliberate  practice  opportunities,  Levels  2  and  3 
training  included  structured  practice  exercises  that  tracked  directly  to  the  material  covered  in  the 
training  and  could  serve  as  checks  on  whether  the  staff  had  acquired  the  skills  taught. 

The  Level  4  training  was  to  be  a  series  of  small  group  structured  exercises  that  provided 
the  first  opportunity  to  practice  the  new  skills  in  a  brief  scenario-based  context.  It  was  designed 
and  developed  to  provide  the  transition  from  system  training  to  staff  process  training.  The  focus 
of  the  Level  4  exercises  was  carried  another  step  beyond  the  Level  3  training,  by  prompting  the 
unit  members  to  think  about  how  they  were  to  woric  together  within  their  node.  No  new  system 
tools  or  functions  were  introduced  during  this  training.  The  exercises  were  structured  so  that 
each  node  would  train  independently  on  the  same  collective  function.  This  structure  allowed  the 
unit  members  to  further  explore  and  understand  their  roles  and  functions  within  each  node,  free 
of  outside  distractions.  It  dso  reinforced  the  multi-functionality  aspect  required  to  operate  in  the 
future  battalion  staff  structure.  These  exercises  were  to  prepare  the  unit  members  to  participate 
in  the  Level  5  TDXs,  which  focused  on  collective  training.  Level  4  training  was  expected  to  run 
3-4  hours.  Also,  during  the  Level  4  training,  the  first  team  training  sessions  were  to  be 
conducted,  introducing  the  staff  members  to  some  considerations  of  information  flow  and  their 
roles  and  functions.  The  team  training  sessions  were  adapted  from  the  staff  process  training 
sessions,  which  are  a  part  of  the  decision-skills  training  (Klein  et  al.,  1996)  described  earlier. 

Finally,  Level  5  training  consisted  of  the  four  TDXs  and  the  embedded  team  training 
sessions.  The  TDXs  brought  together  all  training  participants  (i.e.,  primary  training  audience. 
White  Cell  staff,  company  commanders,  scouts)  to  conduct  collective  structured  exercises  using 
the  full  capabilities  of  the  SC^  system.  The  exercises  provided  the  opportunity  to  practice  new 
skills  in  a  tactical  scenario  similar  to  the  type  of  mission  used  during  Ae  pilot  and  trials.  These 
exercises  not  only  allowed  practice  on  the  system,  but  provided  the  context  for  introduction  of 
tools  based  on  the  NDM  model  and  team  process  research. 

Design  of  the  TDXs  was  based  on  the  performance  analysis  of  the  full  BCR  n  mission 
flow  described  earlier,  from  planning  and  preparation  through  execution  and  sustainment 
activities.  This  analysis  led  to  the  delineation  of  four  distinct  segments  that,  together,  required 
the  participants  to  practice  a  wide  range  of  staff  processes  and  SC^  functions.  TTie  four  segments 
that  were  outlined  are:  a)  mission  analysis  with  wargaming;  b)  mission  rehearsal;  c)  execution  of 
a  squadron  branch;  and  d)  execution  of  a  brigade  sequel. 
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Table  1  links  training  aspects  discussed  in  the  literature  review  that  were  adopted  for 
inclusion  into  the  various  training  levels  in  the  prototype  training  package.  In  addition,  the 
sources  from  which  the  training  aspects  were  derived  are  included. 

Table  1 


Theoretical  Training  Aspects  Linked  to  Prototype  Training  Package  Levels 


Training  Level 

Training  Aspect 

Sample  Theoretical  Sources 

1 

Advance  Organizer 

Cannon-Bowers  et  al.,  1992 
Cannon-Bowers  et  al.,  1998 
Kraiger  et  al.,  1995 

Smith  et  al.,  1997 

West  et  al.,  1991 

2, 3,4 

Part-Task  Training 

Mand  et  al.,  1989 

Means  et  al.,  1993 

Stammers,  1982 

2,  3,4,5 

Deliberate  Practice 

Ericsson  et  al.,  1993 
Frederiksen  &  White,  1989 
Jones,  1989 

Means  et  al.,  1993 

3,4,5 

Context-Based  Training 

Bowers  &  Bell,  1997 
Cannon-Bowers  et  al.,  1992 
Means  et  al.,  1993 

Salisbury,  1990 

4,5 

Decision-Making 

Baron  &  Brown,  1991 

Fallesen  et  al.,  1996 

Klein  et  al.,  1993 

Klein,  McCloskey,  et  al.,  1997 
Zsambok,  1997 

4,5 

Teamwork  Skills 

Carmon-Bowers  et  al.,  1995 
Salas  &  Cannon-Bowers,  1997 
Salas  et  al.,  1997 

Stout  et  al.,  1999 

4,5 

Situational  Awareness 

Endsley,  1988 

Endsley,  1995 

Klein  et  al.,  1996 

Prince  et  al.,  1992 

Entin  et  al.,  1994 

4, 5  Information  Management  Freeman  et  al.,  1997 

Serfaty  &  Entin,  1997 

Note.  Training  Levels;  1  =  Orientation  to  organization  and  processes;  2  =  Fundamental  skills;  3  =  Functional 
skills;  4  =  Individual  and  collective  tasks;  5  =  TDXs  on  staff  processes  and  cognitive  skills. 
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Team  training  sessions  were  also  included  as  an  integral  part  of  the  TDXs.  These 
sessions  were  designed  to  help  the  staff  to  focus  on  their  roles  and  functions  within  and  across 
nodes,  their  shared  situational  awareness,  a  shared  understanding  of  the  conunander’s  intent  and 
their  plan  for  accomplishing  it,  and  their  intensive  review  of  their  own  processes  through  the 
mission  segment.  With  the  team  training  sessions,  it  was  estimated  that  the  TDXs  would  require 
2-3  days. 

As  described  earlier,  the  period  of  time  provided  for  “train  up”  was  five  days.  Figure  2 
shows  the  planned  training  sequence  for  the  Level  1  through  Level  5  training.  As  indicated  in 
the  figure,  Level  2  and  3  training  focused  on  the  individual,  Level  4  progressed  to  a  small  group 
(node)  focus,  and  the  Level  5  TDXs  and  team  training  sessions  brought  all  elements  together  for 
collective  training. 


Day  1 

Day  2 

Day  3 

Day  4 

Days 

Level  1,  Initial 
Orientation 

Levels, 
Functions 
Training  (Cont.) 

Level  .X 

I’DX  Rehe.irsal 

UveI5, 

TDX  4.  Squadron 

Level  2, 
Fundamentals 
Training 

Levd  5,TDX  1, 
Mission  Analysis 
and  Warganiing 

Lev  el  .5. 

TDX  3..  Squadron 
Branch 

Sequel 

Level  3, 
Functions 
Training 

Commander's 

Time 

Note.  I  I  =Individual  training  =Small  group  training  |m  =Collective  training 

Figure  2.  Planned  staff  training  schedule. 


Training  Development 

All  of  the  materials  and  information  associated  with  the  recommendations  for  Level  1 
through  Level  4  are  contained  in  Training  and  Measurement  Support  Package  Volume  2  (1999) 
and  examples  of  the  materials  can  be  seen  in  Appendix  C.  It  is  important  to  note  that,  for  Levels 
1-3,  these  were  detailed  recommendations  with  sample  practice  exercises  rather  than  fully- 
developed  training  products.  They  were  intended  to  be  used  as  templates  for  further 
development.  Levels  4-5  material  included  completed  training  products. 

The  primary  focus  of  the  Level  1  training  was  to  orient  the  training  audience  to  what  to 
expect  over  the  next  three  weeks.  The  recommendations  included  draft  slides  for  the  information 
briefing  that  would  be  delivered  by  the  Training  Director.  The  information  contained  in  the  draft 
briefing  slides  covered: 

•  Purpose  and  Background 

•  Data  Collection 

•  Training  Site  Conditions 

•  Training  Expectation 

•  Training  Schedule 
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Within  the  briefing,  the  new  information  presented  during  training  was  linked  to  familiar 
procedures.  This  included  a  comparison  of  the  unique  future  battle  staff  features  to  current 
analog  procedures  and  topics  such  as  staff  structure,  equipment,  planning  processes,  and  METT- 
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The  Level  2  and  Level  3  training  package  was  directed  at  the  primary  system  trainers,  the 
RAs.  The  materials  included  training  objectives,  guidance  on  how  the  training  should  be 
conducted,  and  training  plan  outlines.  These  outlines  were  step-by-step  listings  of  the 
information  being  presented  to  the  unit  members,  with  special  notes  to  the  RAs  concerning 
information  to  include  in  the  delivery  of  the  training.  The  instructions  indicated  whether  a 
particular  tool  or  information  had  been  previously  introduced  so  that  its  relevance  to  a  functional 
topic  could  be  reinforced  before  proceeding  with  the  new  information  on  that  topic.  Each  outline 
prescribed  a  sequence  for  the  training  content,  although  they  did  not  detail  all  of  the  steps  in  SC 
system  operation.  By  completing  this  outline,  future  staff  trainers  could  convert  the  training  plan 
outlines  to  full  training  plans. 

To  address  the  issue  of  providing  sufficient  practice  using  the  newly  acquired  SC^  system 
skills,  structured  practice  exercises  were  also  recommended.  These  exercises  consisted  of  a 
series  of  statements,  or  cues,  to  prompt  specific  actions  from  the  training  audience.  Each 
structured  practice  exercise  covered  a  30-minute  segment  of  the  training  plan  outline  content, 
and  matched  the  sequence  of  actions  presented  in  it.  Within  the  exercises,  the  steps  were 
repeated  two  to  three  times  as  the  guidance  provided  by  the  RAs  was  reduced  to  increase 
difficulty.  For  Levels  2  and  3,  sample  structured  practice  exercises  were  provided  as  a  template 
from  which  the  remaining  exercises  could  be  completed. 

Level  2  training  provided  for  simple  instmction  in  node  operations,  crewman’s  access 
unit  (CAU)  radio  and  intercom  use,  and  basic  operation  of  the  plan  view  display  (PVD), 
whiteboard,  and  e-mail.  Level  3  training  then  showed  participants  how  to  use  &e  more  advanced 
tools.  Both  training  participants  and  SCT  functions  were  grouped  during  Level  3  training,  so  that 
specific  content  was  addressed  to  the  specific  users.  The  functional  groupings  that  resulted  from 
the  system  tool  analysis  were  as  follows: 

•  Common  relevant  picture  management; 

•  Conference  management; 

•  Information  management; 

•  Orders  production;  and 

•  Other  support  functions  (e.g.,  unmanned  aerial  vehicle  [UAV]  and  battlefield  planning 

visualization  [BPV]). 

For  Level  3  training,  use  of  some  tools  and  functions  (e.g.,  viewing  snail  displays  and 
getting  reports  of  enemy  activity)  required  having  virtual  or  constructive  vehicles  moving  on  the 
battlefield.  Other  functions  (e.g.,  bringing  up  saved  overlays  and  forwarding  orders)  required 
that  prepared  tactical  products  be  available  to  participants.  Scenario  products  fipom  the  BCR  n 
were  pulled  from  archives  by  the  MMBL  and  stored  on  the  SC^  system  to  present  the  needed 
conditions. 


Level  4  training  built  on  the  unit’s  basic  SC'*  knowledge  by  providing  them  the 
opportunity  to  practice  their  functional  system  operation  skills  in  a  scenario-based  context  within 
their  own  nodes.  No  new  system  tools  or  functions  were  introduced;  rather,  the  training 
consisted  solely  of  structur^  practice  exercises.  The  exercises  focused  on  selected  collective 
functions  (determined  from  the  initial  analysis)  that  would  be  performed  by  small  groups  within 
the  unit  during  battle  staff  training,  and  included: 

•  Commander’s  guidance 

•  Develop  CO  A 

•  Order  development 

•  Development  of  annex  to  an  order 

Because  the  Level  4  training  was  designed  to  be  structured  and  scenario-embedded,  a  full 
TSP  was  developed  for  the  training,  rather  than  simply  providing  reconunendations.  The  TSP 
included: 

•  ModSAF  exercise  files  containing  starting  conditions  and  some  scenario  events; 

•  scenario  event  guides  for  use  by  the  White  Cell; 

•  scripted  whiteboard  messages  from  the  Higher  Headquarters; 

•  the  Higher  Headquarters  Order  with  the  Effects  and  Reconnaissance  Surveillance  Target 

Acquisition  (RSTA)  Annexes;  and 

•  the  Higher  Headquarters  operations  overlay. 

The  training  for  the  primary  training  audience  was  to  be  conducted  under  the  direction  of 
the  Training  Director,  with  RAs  assisting  at  the  nodes.  Level  4  training  was  to  be  initiated  with 
the  transmission  of  the  Higher  Headquarters  Order  with  Effects  and  RSTA  Annexes.  Each  of  the 
four  nodes  was  to  produce  a  specific  product.  As  each  node  completed  the  output  for  each 
exercise,  it  was  to  be  transmitted  to  the  Command  1  node.  The  Squadron  Commander  would 
then  initiate  a  conference  with  the  other  node  officers-in-charge  (OICs)  to  discuss  the  output 
result  and  provide  guidance  on  his  expectations  as  to  content  and  format.  When  the  Commander 
completed  his  review,  the  output  would  be  sent  to  the  other  nodes  to  begin  the  next  exercise. 

During  the  structured  practice  exercises,  the  extended  training  participants  also  were  to 
receive  training  using  the  same  exercise  files.  This  training  was  to  focus  primarily  on  a  series  of 
maneuver  exercises  followed  by  an  engagement.  The  two  groups,  primary  and  extended 
participants,  were  also  to  be  brought  together  for  one  combined  activity  during  the  Develop  COA 
exercise.  This  was  done  to  ensure  the  extended  training  participants,  particularly  the  company 
commanders,  were  included  in  the  decision-making  and  team  process. 

The  training  for  the  extended  training  participants  was  to  begin  when  an  order  to  conduct 
a  terrain  reconnaissance  was  transmitted  to  the  company  commanders  under  the  direction  of  the 
Training  Director,  or  his  designee.  The  subsequent  exercises  would  then  be  initiated  by  a  similar 
electronic  transmission  of  orders.  Again,  RAs  were  to  assist  with  system  operations  at  the  nodes. 
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Another  component  of  the  Level  4  training  was  the  introduction  of  the  staff  process 
training.  This  training  was  designed  to  involve  two  one-hour  sessions,  one  on  Information 
Management  and  another  on  Roles  and  Functions.  The  training  sessions  were  to  be  conducted  in 
a  classroom  setting.  A  series  of  slides  and  activities  were  developed  that  detailed  the  information 
found  in  Table  2. 

Table  2 


Level  4  Team  Training  Session  Descriptions 


Training  Session 

Information  Provided 

Information  Management 

Generic  “information  management”  exercise  to  highlight  difficulties 
Required  knowledge  for  information  management 

Model  of  information  management 

Errors  in  information  management 

Low  level  strategies  and  solutions  (e.g.,  e-mail  formatting  protocols) 

High  level  strategies  and  solutions  (e.g.,  understanding  commander’s 
intent) 

Roles  and  Functions 

Short  lecture  on  the  importance  of  roles  and  functions 

Connections  between  roles  and  functions  and  decision-making, 
coordination,  and  information  management 

Discussion  of  how  the  unit  members  perceive  their  roles  and  functions 
for  the  upcoming  BCR  HI 

Format  for  the  unit  to  use  to  self-correct  during  the  exercise  if  the  need 
arises 

The  TDXs,  Level  5  training,  were  the  culmination  of  the  squadron  level  train-up  for  the 
BCR  in.  All  of  the  materials  and  information  for  Level  5  are  contained  in  Training  and 
Measurement  Support  Package  Volume  3  (1999)  and  examples  of  the  materials  can  be  seen  in 
Appendix  D.  WMe  the  initial  training  was  to  focus  primarily  on  individual  skills  and  functions 
within  the  node,  the  TDXs  were  designed  to  provide  a  battalion  or  squadron  battle  staff  the 
opportunity  to  apply  these  skills  and  functions  to  accomplish  collective  battle  staff  tasks  using 
SCr  tools.  The  Training  Director  would  control  the  flow  of  the  TDXs.  The  RAs,  members  of 
the  White  Cell,  and  OPFOR  Controller  would  provide  support  but  would  not  be  in  control  of  the 
TDX  training. 

The  four  TDXs  shared  a  common  tactical  scenario  and  were  projected  to  last  4-8  hours 
each.  The  events  paralleled  situations  contained  in  the  trials.  They  were  designed  to  assist  the 
squadron  participants  in  organizing  and  performing  common  battle  staff  collective  tasks. 
Descriptions  of  tihe  TDXs  are  provided  in  Table  3. 
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Table  3 


TDX  Objective  Descriptions 


TDX 

Objective 

Mission  Analysis 
and  Wargaming 

Provided  the  battle  staff  practice  on  the  decision-making  process.  The  battalion 
or  squadron  conducted  a  roadmarch  based  on  a  movement  order  provided  in  the 
tactical  materials.  As  they  conducted  their  movement,  they  received  an  order, 
intelligence  summaries  (INTSUMs),  and  defensive  battle  force  order.  The 
battle  staff  developed  and  issued  its  order  as  it  completed  its  movement, 
occupied  an  assembly  area  (AA),  and  established  a  screen  line  for  security. 

Rehearsal 

Provided  the  battle  staff  and  company  commanders  practice  conducting  a 
virtual  rehearsal.  The  squadron  completed  its  preparations  within  the  AA  and 
rehearsed  the  plan  developed  during  TDX  1.  Near  the  end  of  the  exercise,  the 
squadron  received  an  INTSUM  that  OPFOR  reconnaissance  had  crossed  the 
international  border  (IB),  and  was  instructed  to  establish  its  defense. 

Execution  of  a 
Battalion  or 
Squadron  Branch 

Provided  the  battle  staff  practice  in  modifying  and  executing  a  branch  to  an 
existing  plan  during  operations.  As  the  squadron  moved  to  its  defensive  sector, 
it  encountered  and  was  to  destroy  enemy  reconnaissance  elements.  The 
exercise  ended  with  the  squadron  occupying  its  defensive  sector. 

Execution  of  a 
Brigade  Sequel 

Provided  the  battle  staff  practice  with  planning  a  future  operation  while 
executing  a  current  mission.  As  the  squadron  executed  its  defense  to  defeat  a 
forward  detachment.  Higher  Headquarters  issued  fragmentary  order  (FRAGO) 
to  its  defensive  battle  force  order.  The  FRAGO  required  the  2  Squadron  to 
conduct  an  on  order  (0/0)  counterattack  to  defeat  a  first  echelon  motorized 
infantry  brigade  (MIBR)  from  the  flank. 

The  materials  developed  in  support  of  the  TDXs  were  based  on  the  methodology  for 
development  of  structured  simulation  based  training  (Campbell  et  al.,  1995;  Campbell  &  Deter, 
1997;  Flynn  et  al.,  1998).  The  materials  included: 

•  Overview 

•  Event  Description 

•  Event  Guide 

•  OPFOR  and  Blue  Forces  (BLUFOR)  Workstation  Guideline 

•  Tactical  Materials 

•  Plan  Sheets 

•  Training  Director  Guidelines 

Embedded  in  each  TDX  were  three  team  training  sessions  that  focused  on  the  unit’s 
decision-making  and  team  functioning  skills.  The  description  of  each  training  session  is  shown 
in  Table  4.  The  team  training  sessions  were  to  be  initially  facilitated  primarily  by  the  Training 
Director.  The  intent  for  these  sessions  was  to  have  the  Squadron  Commander  take  over 
responsibility  in  leading  these  training  exercises  as  the  TDXs  progressed. 
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Table  4 


Level  5  Team  Training  Session  Descriptions 


Team  Training 
Session 

Objective 

Pre-Action  Analysis 

Focus  staff  attention  on  shared  understandings  of  roles  and  functions, 
information  management,  workload,  and  situational  awareness.  The  purpose  is 
to  anticipate  potential  problem  areas  and  help  generate  how  they  will  solve 
problems  should  they  arise.  These  sessions  will  also  allow  time  to  review  those 
action  items  identified  in  previous  debriefs  for  sustainment  or  improvement. 

Commander’s 

Timeout 

Recalibrate  the  team  to  the  Commander’s  situational  awareness.  The  benefits  of 
performing  a  Commander’s  Timeout  are  that  it  causes  the  Commander  to  stop 
and  think  about  his  situational  awareness  and  to  communicate  that  to  the  staff.  It 
allows  the  Commander  to  gain  information  to  questions  he  may  have  and  help 
the  team  to  focus  on  keeping  the  Commander  aware  of  the  ground  truth. 

Team  Decision- 
Making  Debrief 

Focus  on  issues  related  to  team  processes,  decision-making,  roles  and  functions, 
and  information  management.  The  team  will  also  identify  action  items  for 
sustainment  and  improvement. 

These  sessions  were  designed  to  train  information  management,  decision  making  and 
teamwork  skills.  The  sessions  were  to  occur  before,  during  and  after  each  exercise,  respectively. 
The  pre-action  analysis  was  to  begin  with  the  exercise  director  explaining  the  tactical  situation 
for  the  exercise.  Then  the  commander  would  explain  his  goals  for  the  exercise.  Each  of  the 
node  OICs  would  then  discuss  the  node’s  role  during  the  exercise.  Finally,  each  staff  member 
would  list  one  challenge  or  show  stopper  that  could  occur  during  the  exercise.  These  would  be 
discussed  and  ways  to  deal  with  the  challenges  developed.  The  commander’s  timeout  was  to 
occur  during  the  exercise  at  a  point  of  uncertainty  caused  either  by  conditions  of  the  situation  or 
staff.  The  commander  would  explain  what  is  happening,  what  information  is  needed,  immediate 
goals,  and  what  will  occur  in  30  to  60  minutes.  The  team  decision-making  debrief  was  to  occur 
after  the  exercise.  First  the  commander  would  discuss  a  difficult  decision  made  during  the 
exercise,  what  information  was  available  and  what  information  was  not.  Each  node  QIC  would 
then  discuss  their  node’s  role  during  that  decision.  Finally,  node  OICs  would  discuss  their  own 
difficult  decisions. 

The  team  training  sessions  build  on  the  Klein,  McCloskey,  et  al.  (1997)  Decision  Skills 
Training  program  described  earlier  in  this  report.  The  Decision  Skills  Training  program  was 
focused  on  improving  individual  decision-making  skills.  While  maintaining  the  focus  on 
decision-making,  the  team  training  sessions  are  broader  in  scope  to  focus  on  training  decision¬ 
making  skills  in  an  Army  battalion-level  staff  operating  as  geographically  separated  teams  as  it 
plans,  prepares,  and  executes  tactical  missions. 

Performance  Evaluation  Methods 

Embedded  in  the  requirement  to  design  and  develop  a  prototype  training  package  for 
future  battle  staffs  is  the  ne^  to  evaluate  the  performance  of  the  staff  after  it  has  received  the 
training.  Existing  standards  for  measuring  the  performance  of  battalion-level  staffs  (DA,  1988) 
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were  reviewed  and  deemed  to  have  very  little  direct  application  to  future  staff  training  evaluation 
since  they  are  based  on  a  staff  organization  and  analog  decision-making  process  that  would  not 
be  used  during  the  BCR  HI.  Draft  training  and  evaluation  outline  standards  incorporating  current 
battalion-level  digital  technology  at  the  battalion  level  (U.  S.  Army  Armor  Center  [USAARMC], 
1997)  were  also  reviewed  and  not  adopted  for  the  same  reasons.  As  a  starting  point  for 
evaluation,  the  Battle  Lab  Experiment  Plan  (BLEP)  for  the  BCR  in  (Mounted  Maneuver 
Battlespace  Lab,  1998b)  identified  several  issues  related  to  advanced  digitization’s  effects  on 
battle  command  at  brigade  and  below*  that  could  form  the  basis  for  developing  future  staff 
training  performance  standards.  The  central  issue  was:  If  the  commander  is  provided  timely, 
accurate,  decision-centered  information  that  allows  him  to  visualize  the  battlefield,  how  is  the 
commander’s  ability  to  conduct  battle  command  improved? 

Earlier  BCRs  demonstrated  the  potential  for  value  to  be  added  to  battle  command  by 
using  advanced  digital  C'*!  systems.  However,  the  BCRs  did  not  focus  on  the  commander’s  need 
to  visualize  the  battlefield  and  the  staff  organization  and  process  that  would  support  him.  The 
BCR  in  would  provide  this  focus  by  examining  battlefield  visualization  and  ways  to  improve  it 
through  the  use  of  more  complete  information  flow,  more  optimal  structuring  of  the  staff,  and 
training  in  ways  to  optimize  use  of  the  information.  The  three  issues  of  interest  were: 

•  How  effectively  does  the  objective  C^I  system  enable  the  commander  to  visualize  the 
battlefield? 

•  What  efficiencies  in  speed,  knowledge,  and  manning  requirements  are  gained  from 
reengineered  battle  command? 

•  What  are  the  major  impacts  of  reengineered  battle  command  across  the  spectrum  of  doctrine, 
training,  leader  development,  organizations,  materiel,  and  soldiers  (DTLOMS)? 

The  BCR  HI  hypotheses  were  based  on  six  overarching  hypotheses: 

•  The  objective  C^I  system  will  provide  timely,  required  critical  information  requirements. 

•  The  objective  C^I  system  will  provide  the  necessary  capabilities  for  the  key  nodes  to  meet  the 
commander’s  critical  information  requirements  (CCIRs). 

•  The  revised  MDMP  will  enhance  the  span  of  control  in  tactical  operations. 

•  The  objective  C^I  system  will  enhance  synchronization  of  assets. 

•  The  objective  C'*!  system  will  provide  information  to  assist  the  commander’s  decisions. 

•  The  objective  C^I  system  will  facilitate  multifunctionality. 

Since  the  future  battalion  battle  staff  model  that  the  project  team  was  using  to  develop  the 
training  did  not  have  published  doctrine  or  tactics,  techniques,  and  procedures  upon  which  to 
base  training  and  performance  evaluations,  the  MMBL  issues  were  used  as  a  starting  point. 

A  total  of  10  questions,  derived  from  the  three  MMBL  issues,  were  addressed  by  the 
performance  measures.  The  measurement  would  be  multifaceted,  involving  a  combination  of 
automated  measures,  surveys,  observations,  and  interviews.  Specifications  for  the  evaluation 


’  Information  for  this  section  of  the  report  has  been  adapted  from  the  BLEP  (MMBL,  1998b). 
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plan  began  with  analysis  of  the  questions  and  determination  of  the  methods  that  could  best  be 
used  to  address  each  question.  The  specific  measures  (e.g.,  survey  and  interview  questions, 
observation  points)  were  then  designed  and  developed  to  provide  broad  yet  efficient  coverage  of 
the  issues. 

Measures  Analysis 

The  goal  was  to  address  each  question  by  at  least  two  methods,  to  increase  the  reliability 
and  validity  of  the  data  and  interpretations.  Additionally,  each  method  has  recognized  strengths 
and  weaknesses.  For  example,  interviews  can  potentially  provide  a  wealth  of  information  on 
various  questions,  as  participants  explore  issues  either  individually  or  in  a  group;  however,  they 
are  time-consuming  to  conduct  and  difficult  to  analyze  and  summarize.  Survey  data  are  more 
easily  obtained,  requiring  less  participant  time,  but  do  not  usually  yield  lengthy  explanations  or 
comments.  The  intent  was  to  exploit  the  method  strengths  but  not  compromise  the  utility  of  the 
data-collection  opportunities.  Table  5  portrays  the  matrix  of  research  questions  and  the  measures 
used  to  evaluate  them. 

The  use  of  automated  measures  was  focused  on  the  first  six  MMBL  issues.  Although  the 
target  variables  in  the  MMBL  issues  (e.g.,  efficient  synchronization,  span  of  control)  were  not 
directly  measurable  by  objective  data,  simulation-prt^uced  measures  could  provide  partial 
information  on  the  issues.  The  automated  measures  were  gathered  by  the  MMBL’s  Data 
Collection  and  Analysis  System  (DCA)  during  the  virtual  simulation  used  in  the  BCR  HI.  The 
DCA  is  a  set  of  tools  designed  to  collect,  reduce  and  analyze  battlefield  performance,  command 
and  control,  communications,  and  other  types  of  data  in  distributed  simulations. 

Surveys  were  used  for  four  of  the  team  process  and  decision-making  issues.  Within  the 
NDM  model  and  training  design,  the  focus  was  on  three  core  skill  areas:  decision-making, 
information  management,  and  team  performance.  The  surveys  focused  on  perceptions  of 
teammate  roles  and  responsibilities,  workload,  communication  load,  and  process.  It  was 
expected  that  high  performance  teams  would  share  awareness  and  knowledge  of  individual  roles 
in  the  larger  team,  the  distribution  of  load,  and  the  overall  goals  of  the  shared  mission.  While 
improved  situational  awareness  might  foster  team  communication  and  awareness  of  the  activities 
of  others  in  the  team,  it  could  prove  overwhelming,  leading  to  information  overload  and  a  loss  of 
focus  on  the  command  decisions  central  to  the  job.  The  surveys  were  designed  to  begin  to 
achieve  a  greater  understanding  of  these  issues. 

The  observation-based  measures  were  used  to  provide  additional  information  on  five  of 
the  questions  by  focusing  on  qualitative  aspects  of  performance  that  are  hard  to  capture  using 
automated  means.  Five  key  aspects  of  performance  were  designated  for  observation  focus: 

•  Communication:  the  exchange  of  information  between  two  or  more  team  members  in  the 
prescribed  maimer  and  using  proper  terminology. 

•  Monitoring:  how  well  team  members  observed  the  activities  and  performance  of  other 
team  members. 

•  Back-up:  how  well  team  members  assisted  the  performance  of  other  team  members. 
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•  Coordination:  whether  team  members  executed  their  activities  in  a  timely  and  integrated 
manner. 

•  Team  orientation:  the  commitment  exhibited  by  team  members  working  together. 


Table  5 

Questions  Addressed  by  Each  Measure 


1 .  Can  the  Reengineered  Battle  Command  (RBC) 

decrease  the  time  for  planning  and  increase  the  X 

time  to  prepare  and  rehearse? 

2.  Can  the  RBC  provide  the  information  and  support 

system  to  assist  the  Commander’ s  decision-making  X 

process? 

3.  Can  the  RBC  allow  efficient  synchronization  of 

combat,  combat  support,  and  combat  service  X 

support  assets? 

4.  Does  the  RBC  provide  efficient  battle  tracking  and  ^ 

facilitate  precise  execution? 

5.  Does  the  RBC  contribute  to  more  rapid  and  ^ 

efficient  destruction  of  enemy  forces? 

6.  Can  the  RBC  increase  the  span  of  control  of  the  ^ 

Commander? 

7.  What  effective  tactics,  techniques,  and  procedures 
(TTPs)  are  emerging  from  the  RBC  process? 

8.  What  are  the  information  management  demands  on 
multi-functional  officers  and  what  are  the  impacts 
of  these  demands  on  individual  and  team 
performance,  process,  and  training  requirements? 

9.  What  are  the  impacts  of  RBC  on  team  performance 
and  process  and  how  do  new  tools,  capabilities,  and 
roles  change  the  task  force  organizational 
architecture? 

10.  What  are  the  impacts  of  RBC  on  perceived  and 
actual  workload,  the  attribution  of  workload  to 
individual,  team,  and  task  demands,  and  individual 
awareness  of  the  distribution  of  workload  across 
the  team? 


Measure  Types _ 

Survey  Observer  Interview 


X 


XXX 

X  X 

X  X 

X 

X  X 

X 


XXX 


X  X 


X 


The  interviews  were  used  to  provide  information  on  any  issues  not  completely  covered 
by  the  other  measures  as  well  as  to  allow  participants  to  expand  on  their  responses  to  any  open- 
ended  survey  questions.  They  would  be  conducted  by  project  team  members  familiar  with  the 
design  of  the  prototype  training  package  and  the  MNffiL’s  objectives  for  BCR  HI. 
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One  of  the  difficulties  in  designing  specific  measures  for  the  MMBL  issue-based 
questions  was  that  the  questions  had  not  previously  been  addressed,  thus  lacking  both 
instruments  and  baseline  data.  For  those  issues  where  the  interest  was  in  improvements  (e.g., 
“increase  the  time”  or  “contribute  to  more  rapid  destruction”),  measures  in  BCR  in  would  be 
examined  as  a  possible  way  to  establish  a  baseline  for  future  BCRs. 

Measures  Design  and  Development 

The  specific  content  and  format  of  each  of  the  four  types  of  measures  is  described  below. 
Copies  of  the  data  collection  instruments  are  contained  in  Training  and  Measurement  Support 
Package  Volume  4  (1999). 

Automated  measures.  The  automated  measures  were  designed  to  answer  specific  issues 
regarding  the  effectiveness  and  efficiency  of  the  reengineered  Battle  Command.  The  questions, 
the  specific  measures  of  performance  designed  to  address  them,  and  descriptions  of  their 
operational  definitions  can  be  seen  in  Table  6.  Although  these  measures  do  not  tell  a  complete 
story  about  staff  performance  on  their  own  since  they  are  focused  on  outcomes  rather  than 
processes,  they  become  more  informative  with  the  addition  of  the  process-oriented  measures. 

Table  6 

BCR  in  Questions  and  Automated  Measures  of  Performance 


Question 

Measure  (tf  Performance 

Description 

1 .  Can  the  RBC  decrease 
the  time  for  planning 
and  increase  the  time 
to  prepare  and 
rehearse? 

Battalion  staff  communication 
patterns 

Total  time  spent  using  each  of  the 
communication  tools  for  each  node 
position  fcH*  each  planning  and  execution 
session 

2.  Can  the  RBC  provide 
the  information  and 
support  system  to 

UAV  mission  effectiveness 

Number  of  OPFOR  vehicles  first  detected 
by  UAVs  divided  by  total  number  of 

OPFOR  vehicles  detected 

assist  the 

Commander’s 

decision-making 

process? 

Percent  of  enemy  vehicles 
inside  the  battalion’s  area  of 
responsibility  that  were 
detected 

Number  of  unique  OPFOR  vehicles 
detected  by  sensors,  scouts,  or  weapons 
systems  controlled  by  the  battalion  divided 
by  the  number  of  OPFOR  vehicles 

Percent  of  enemy  vehicles 
inside  the  battalion’s  area  of 
responsibility  that  were  never 
detected 

Number  of  unique  OPFOR  vehicles  not 
detected  by  sensors,  scouts,  or  weapons 
systems  controlled  by  the  battalion  divided 
by  the  number  of  OPFOR  vehicles 

Use  of  SC^  conununication 
tools  during  mission 

Total  tinte  spent  using  each 
communication  tool  and  the  number  of 
conununication  tool  initiations  per  mission 
for  each  node  position 

(table  continues) 


38 


Table  6  (Continued) 


Question 

Measure  of  Performance 

Description 

Use  of  Stealth  Control,  Terrain 
Tool,  Field  of  View  Tool, 

Snail  Display,  and  Forward 

Line  of  Troops  Display  during 
mission  planning 

Total  time  spent  using  each  tool  and  the 
number  of  tool  initiations  for  each  node 

3.  Can  the  RBC  allow 
efficient 

Percent  of  OPFOR  kills  inside 
effects  box 

Number  of  OPFOR  kills  in  effects  box 
divided  by  total  number  of  OPFOR  kills 

synchronization  of 
combat,  combat 
support,  and  combat 
service  support 
assets? 

Ratio  of  indirect  to  direct  fire 
OPFOR  kills 

Number  of  indirect  fire  kills  to  number  of 
direct  fire  kills 

4.  Does  the  RBC 
provide  efficient 
battle  tracking  and 

Percent  of  OPFOR  vehicles 
engaged  from  flank  or  rear 

Total  number  of  flank  or  rear  engagements 
on  OPFOR  vehicles  divided  by  total 
number  of  OPFOR  vehicles 

facilitate  precise 
execution? 

Percent  of  BLUFOR  vehicles 
engaged  from  flank  or  rear 

Total  number  of  flank  or  rear  engagements 
on  BLUFOR  vehicles  divided  by  total 
number  of  BLUFOR  vehicles 

Average  range  of  OPFOR  fire 
engagements 

Average  range  of  friendly  weapon  systems, 
by  type,  against  OPFOR  vehicles  that  were 
kill^  during  a  mission 

5.  Does  the  RBC 
contribute  to  more 

OPFOR  vehicle  kills  by 
friendly  weapons  types 

Number  of  OPFOR  vehicle  kills  by 
fiiendly  weapon  types  during  the  mission 

rapid  and  efficient 
destruction  of  enemy 
forces? 

Time  to  destroy  OPFOR 

Time  from  first  OPFOR  engagement  until 
OPFOR  vehicle  losses  exceed  70% 

6.  Can  the  RBC  increase 
the  span  of  control  of 
the  Commander? 

Number  of  subordinate  unit 
leaders  Squadron  Commander 
personally  contacted  during 
mission  execution 

Commander’s  frequency  of  and  amount  of 
time  for  use  of  communications  tools 
across  each  of  the  different  personnel  with 
which  he  interacted  during  mission 
execution 

Note.  Area  of  responsibility  is  delineated  by  its  rear,  flank,  and  forward  boundaries  assigned  by  its  higher 
headquarters. 


Survey  measures.  The  surveys  that  addressed  the  MMBL  issues  were  designed  to  be 
completed  by  the  primary  training  audience  at  the  conclusion  of  the  training  week  and  on  each 
day  of  the  pilot  test  and  trials.  There  were  five  surveys,  as  described  below. 

The  Organizational  Awareness  Survey  was  designed  to  determine  the  extent  to  which 
each  member  of  the  team  is  aware  of  the  roles  and  activities  of  their  teammates  during  mission 
execution.  The  items  on  the  survey  were  to  provide  a  record  of  what  each  individual  thought 
every  other  individual  was  doing  at  a  particular  time.  This  would  lead  to  an  understanding  of 
whether  the  system  of  tools  and  capabilities  introduced  in  BCR  HI  impact  team  process  and 
performance  via  improved  abilities  to  acquire  and  maintain  team-level  situational  awareness. 
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The  Operations  Planning  Survey  was  designed  to  assess  the  extent  to  which  trained 
behaviors  are  practiced  in  the  planning  and  rehearsal  stages  of  each  mission.  The  data  could  help 
to  determine  whether  the  use  of  enhanced  planning  support  tools  provides  the  staff  with 
improved  plan  knowledge  and  preparedness  for  communication  and  coordination,  and  whether 
these  things  impact  individual  understanding  of  roles  in  the  mission. 

The  Information  Management  Survey  was  designed  to  track  the  information  management 
demands  of  each  mission  and  collect  data  on  the  information  management  behaviors  practiced 
during  each  mission.  By  collecting  these  data,  the  match  between  participants’  perceptions  of 
the  sources  of  information  management  load  and  the  actual  sources  could  be  assessed.  The 
major  issues  to  be  investigated  through  this  instrument  centered  on  the  extent  to  which  the 
system  of  tools  supported  manageable,  effective  communication  and  coordination  as  well  as 
awareness  of  the  communication  loads  imposed  on  the  team.  With  so  many  options  to  choose 
from,  a  whole  new  class  of  decisions  is  introduced  by  the  communication  tools  provided.  The 
degree  to  which  individuals  are  able  to  track  and  predict  information  load,  choose  the  right  tools 
for  the  job,  and  coordinate  effectively  without  information  overload  are  all  empirical  questions. 
Answers  to  these  questions  may  provide  a  foundation  for  standing  operating  procedures  (SOPs) 
or  TTPs  in  future  BCRs. 

The  Individual  and  Team  Workload  Survey^  was  designed  to  assess  individual  workload 
as  well  as  individuals’  perceptions  of  the  workload  faced  by  teammates  in  other  nodes.  Team 
workload  measures  are  predictors  of  team  performance  and  may  provide  insight  into  the  impact 
of  training  as  well  as  the  ability  of  the  system  to  support  awareness  of  the  rest  of  the  team.  The 
basic  question  to  be  answered  was  whether  the  system  of  tools  introduced  in  BCR  m  enhanced 
the  abilities  of  the  staff  to  appreciate  and  maintain  situational  awareness  of  the  workload  across 
the  staff  and  whether  this  ability  changed  with  practice  as  a  team. 

The  Communication  Load  Survey  was  designed  to  assess  the  degree  to  which 
participants’  perceptions  of  information  management  load  across  the  team  match  to  objective 
data.  The  accuracy  of  such  perceptions  is  important  because  those  perceptions  provide  the  basis 
for  individual  decisions  to  send  information,  request  information,  and  time  communications.  It 
was  expected  that  performance  would  be  related  to  the  ability  to  predict  the  information 
management  load  of  others. 

Observations.  The  observation  data  were  time-stamped  information  elements  collected 
on  laptops  by  trained  data  collection  observers.  To  assist  the  observers  with  this  effort,  the 
Observer’s  Data  Collection  fristmment  (ODCI),  a  software  program  created  with  Microsoft 
Visual  Basic®,  was  developed.  By  entering  a  last  name,  the  name  of  the  exercise,  and  the  node 
being  evaluated,  the  observer  was  then  led  into  the  specific  events  that  would  cue  the  desired 
behavior.  Bach  event  was  described  in  detail  and  a  picture  of  the  graphic  control  measures  used 
during  the  event  was  also  provided.  Data  collection  observers  also  had  space  to  provide 
comments  on  any  of  the  behaviors.  At  the  conclusion  of  the  exercise,  the  observer  ratings  and 
comments  were  stored  in  a  word  document  file  that  could  readily  be  downloaded  for  analysis  and 
feedback.  Sample  screens  and  output  documents  can  be  seen  in  Appendix  E. 


^  Adapted  from  NASA-Ames  Research  Center,  Human  Performance  Group.  (1986).  Collecting  NASA  workload 
ratings:  A  paner-and-pencil  package  rVersion  2.1).  Moffet  Field,  CA:  NASA-Ames  Research  Center. 
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The  information  collected  was  based  on  the  five  teamwork  behaviors  described  earlier. 
Each  behavior  was  observed  and  rated  on  a  7-point  scale.  Data  collection  observers  also  had 
space  to  provide  comments  on  any  of  the  behaviors.  The  specific  questions  for  each  of  the  five 
behaviors  measured  are  provided  in  Table  7.  The  measures  were  collected  for  the  planning  and 
execution  phases  of  each  mission.  In  addition  to  these  five  behaviors,  observers  also  rated  the 
overall  team  performance,  both  for  the  node  they  were  observing  and  for  the  Squadron  as  a 
whole. 

Table  7 

Questions  Addressed  by  Observation  Data 

Behavior _ Questions _ 

Communication  1 .  To  what  extent  were  errors  caused  by  inadequate  within-node  communication? 

2.  To  what  extent  were  errors  caused  by  inadequate  team  conamunication? 

3.  To  what  extent  did  node  members  provide  relevant  information  to  another  node 
member,  in  a  pro-active  way,  without  that  node  member  having  to  ask  for  it? 

4.  To  what  extent  did  the  node  provide  relevant  information  to  other  nodes,  in  a 

pro-active  way,  without  the  other  nodes  having  to  ask  for  it?  _ 

Monitoring  1 .  To  what  extent  did  node  members  alert  each  other  to  impending  decisions  and 

actions? 

2.  To  what  extent  did  the  node  alert  other  nodes  to  impending  decisions  and 
_ actions?  _ 

Back-up  1 .  To  what  extent  did  node  members  anticipate  the  need  to  provide  assistance  to 

other  node  members? 

2.  To  what  extent  did  the  node  anticipate  the  need  to  provide  assistance  to  other 
nodes? 

3.  Did  the  node  members  adjust  individual  tasks  and  responsibilities  to  prevent 
overload? 

4.  Did  the  node  adjust  individual  tasks  and  responsibilities  to  prevent  team 

overload?  _ _ 

Coordination  1 .  To  what  extent  was  the  behavior  of  individuals  in  the  node  coordinated? 

2.  To  what  extent  was  the  overall  team’s  behavior  coordinated? _ 

Team  Orientation  1.  How  congruent/similar  were  individual  node  members’  understandings  of  their 
role  in  the  mission? 

2.  How  congruent/similar  were  node  members’  understandings  of  the  role  of  the 

node  in  the  mission?  _ _ _ 

Note.  Each  question  was  rated  on  a  7-point  scale. 

Interviews.  Protocols  were  prepared  for  structured  interviews  to  be  conducted  with 
individuals  and  small  groups,  consisting  of  the  observers  and  training  audience  members.  The 
interviews  were  conducted  on  the  last  day  of  the  BCR  HI,  and  the  actual  protocols  are  presented 
in  Training  and  Measurement  Support  Package  Volume  4  (1999).  Within  the  training  audience, 
interviews  were  scheduled  with  the  following; 
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•  Commander 

•  Deputy  Commander 

•  Effects  Officer  and  Enemy  Operations  Officer 

•  Control  Vehicle  Battle  Captains 

•  Enemy  operations  personnel  from  Command  2,  Control  1,  and  Control  2 

•  Friendly  operations  personnel  from  Command  2,  Control  1,  and  Control  2 

•  Company  commanders 

The  training  audience  was  segmented  into  groups  of  individuals  with  similar  tasks  and 
responsibilities  in  order  to  encourage  an  open  conversation  and  enable  data  collection  from 
different  perspectives.  Each  group  was  made  up  of  personnel  with  the  same  ranks  in  order  for  all 
group  members  to  be  able  to  express  their  opinions  without  the  discussion  being  dominated  by 
the  highest-ranking  individual.  The  time  for  each  interview  was  limited  to  1  hour.  Because  of 
the  restricted  time  available,  the  scope  of  each  group’s  interview  was  limited  to  focus  on  insights 
that  were  not  obtained  by  survey,  observation,  or  automated  data  collection  methods. 

Pilot  Implementation  of  the  Prototype  Training  and  Evaluation  Package 

As  described  previously,  parallel  to  the  early  conceptualization  of  the  training  need  on 
which  this  project  is  based,  the  MMBL  was  in  the  initial  stages  of  planning  for  their  BCR  HI.  (A 
thorough  description  of  the  BCR  in  design  appears  in  Battle  Lab  Experiment  Final  Report 
[BLEFRJfor  Battle  Command  Reengineering.  Phase  III  [1999]).  It  was  known  from  the  results 
of  the  earlier  BCRs  that  participants  needed  more  structured  training  on  operating  the  digital 
system  and  operating  within  a  reengineered  staff  structure.  An  intact  squadron  staff  was 
scheduled  to  participate  in  the  BCR  m,  so  that  there  would  not  be  the  problem  of  providing 
entry-level  staff  training.  Thus  all  of  the  assumptions  underlying  the  general  training 
recommendation  for  the  prototype  package  were  satisfied.  The  ARI  and  the  MMBL  agreed  to 
work  as  a  team  to  effect  mutual  support  of  each  others’  goals. 

Therefore,  the  general  training  design  was  tailored  for  use  in  the  BCR  HI  which  would 
allow  for  a  pilot  implementation  using  MMBL  personnel  as  the  trainers.  An  added  benefit  was 
ARI  would  be  able  to  conduct  a  formative  evaluation  of  the  prototype  training  package.  The 
specific  design  characteristics  of  the  BCR  DI  which  the  project  team  considered  in  tailoring  the 
prototype  training  package  are  described  below.  The  description  covers  virtual  and  constructive 
simulators,  the  scenario  conditions,  training  audience,  conunander  and  staff  organization,  and 
training  timeline. 

BCR  in  Simulators 

The  BCR  Wl  used  emulation  as  well  as  constructive  and  virtual  simulators.  Figure  3 
shows  the  sample  layout  of  the  SC*  system  hardware  for  the  BCR  m.  The  Effects  or  Enemy 
Operations  Officer  is  positioned  on  the  left.  The  node  officer  in  charge  (OIC)  is  seated  in  the 
center,  with  the  Friendly  Operations  Officer  or  NCO  seated  on  the  right.  The  SC^  system 
included  the  following  capabilities; 
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•  A-  Command  and  Control  Plan  View  Display,  represented  by  the  ModS  AF  two  dimensional 
PVD,  On  the  PVD,  Node  OICs  are  able  to  view  movements  of  all  of  their  own  systems,  as 
well  as  any  OPFOR  units  detected  by  satellite  or  other  sensors.  Overlays  can  be  drawn  on 
the  PVD,  users  can  add  labels  or  other  notes,  and  there  are  tools  that  show  past  events  and 
project  future  movements. 

•  A-  Digitized  modified  combined  obstacle  overlay  (MCOO),  produced  automatically  for  the 
flat  panel  display,  rather  than  as  a  manually  produced  intelligence  overlay. 

•  B-  Flat  Panel  Display,  providing  a  3-dimensional  representation  of  the  battlefield  with  all  of 
the  systems  that  are  visible  on  the  PVD  (i.e.,  friendly  and  detected  OPFOR). 

•  C-  Unmanned  Aerial  Vehicle  (UAV)  sensor  display,  with  video  footage  of  area  currently 
under  surveillance  by  UAV. 

•  D-  Utility  Display  with  video  teleconference  (VTC)  capability  linking  the  commander  and 
his  staff. 

•  D-  Collaborative  whiteboard  capability,  to  allow  the  commander  to  present  his  intent  and 
guidance  to  the  staff  visually  and  quickly.  Users  who  are  part  of  the  whiteboard  session  can 
show  snapshots  from  their  PVDs,  draw  in  different  colors  on  those  images,  add  clipart-style 
labels  and  icons,  and  type  words  onto  the  whiteboard. 

•  E-  Battlefield  Planning  and  Visualization  (BPV)  ,  a  system  that  allows  the  commander  to 
plan,  rehearse,  and  monitor  execution  of  missions  through  a  2-dimensional/3-dimensional 
interface. 

•  E-  Satellite  imagery,  acting  as  the  electro-optic  satellite  sensor  to  deliver  a  direct  downlink 
imagery  feed. 


Note.  Reproduced  from  Battle  Command  Reengineering  III  User’s  Manual  for  the  Battle  Command  Reengineering 
III  Battle  Lab  Watfighting  Experiment  (Mounted  Maneuver  Battlespace  Lab,  1999a). 

Figure  3.  SC^  system  hardware  layout  for  BCR  DDL 
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Vehicles  and  weapon  systems  were  represented  in  either  constructive  or  virtual 
simulation.  Constructive  simulation  (ModSAPO  was  used  to  generate  and  control  the  OPFOR, 
friendly  forces  below  the  company  level,  and  unmanned  vehicles  replicating  both  aerial  and 
ground  sensors  (referred  to  as  UAVs  and  unmanned  ground  vehicles  (UGVs),  respectively). 
Constructive  simulation  workstations  were  used  by  the  fire  support  element  commander,  forward 
support  company  commander,  two  of  the  maneuver  company  commanders,  and  the  three  platoon 
leaders  of  another  maneuver  company. 

In  the  virtual  environment,  simulators  were  used  to  represent  several  vehicles.  These 
included  the  battalion  commander  and  deputy  commander  vehicles  which  were  represented  by 
the  Advanced  Research  Projects  Agency  (ARP A)  Reconfigurable  Simulator  Initiative  (ARSI) 
simulator  and  an  ARSI  mockup,  respectively;  staff  operations  vehicles  (SOVs)  which  were 
represented  by  command  and  control  vehicle  (C2V)  mockups;  and  scout  vehicles  which  were 
represented  by  Future  Scout/Cavalry  System  mockups.  The  virtual  and  constructive 
environments  were  linked  by  means  of  distributed  interactive  simulation  (DIS)  to  form  the 
seamless  battlefield  environment  for  the  participants. 

Trial  Scenario  Conditions 

For  the  BCR  HI,  the  scenario  covered  a  tactical  movement  and  six  mission  segments, 
each  requiring  approximately  one  day.  The  sequence  was  as  follows: 

1.  Tactical  movement  and  defense:  The  unit  conducted  a  tactical  movement  from  its  initial 
location  to  an  assembly  area  where  it  received  an  order  to  defend.  The  rest  of  the  day 
was  spent  planning  and  preparing  for  the  defense. 

2.  Defense:  The  unit  executed  the  defense  that  it  had  planned  and  prepared  against  an 
OPFOR  Motorized  Rifle  Regiment  (soon  to  be  called  a  Mechanized  Infantry  Brigade). 

3.  Movement  to  Engagement:  The  unit  moved  to  engage  an  OPFOR  Motorized  Rifle 
Battalion  operating  as  a  flank  guard. 

4.  Defense:  The  unit  defended  against  an  OPFOR  second  echelon  Motorized  Rifle 
Regiment. 

5.  Movement  to  Engagement:  The  unit  moved  to  engage  an  OPFOR  tank  battalion 
maneuvering  as  a  forward  detachment 

6.  Defense:  The  unit  defended  against  an  OPFOR  Motorized  Rifle  Regiment  from  a  Second 
Echelon  Division. 

7.  Movement  to  Engagement:  The  unit  moved  to  engage  a  Motorized  Rifle  Battalion 
attempting  to  link  up  with  an  air  assault  force. 

For  each  mission,  the  unit  readiness  conditions  were  reset  within  the  simulation.  Thus, 
the  battalion  staff’s  combat  service  support  (CSS)  activity  did  not  have  any  real  impact  from  one 
mission  to  the  next.  A  single  integrated  storyline  was  constructed  to  underlie  the  missions. 

Both  the  fiiendly  and  enemy  forces  were  configured  as  nontraditional  future  forces,  in 
terms  of  equipment,  organization,  and  doctrine.  The  enemy  forces  comprised  a  regimental-sized 
Heavy-Light  Maneuver  Group.  The  MMBL  evaluated  several  friendly  forces  concepts  by  means 
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of  simulation  modeling  between  November  98  and  January  99  to  determine  which  concept  was 
best  able  to  defeat  the  enemy.  Results  of  that  evaluation  are  reported  in  Battle  Lab  Experiment 
Final  Report  (BLEFR)for  Battle  Command  Reengineering,  Phase  77/(1999). 

In  addition  to  the  six  mission  segments,  there  were  also  six  planning  segments  for 
missions  that  would  not  be  executed.  This  concurrent  planning  and  conduct  of  missions  would 
allow  the  MMBL  to  examine  and  refine  the  battalion  organizational  structure  for  current  and 
future  operations. 

BCR  in  Participants 

The  primary  training  audience  (14  members)  comprised  the  commander  and  selected 
staff  of  an  active  Army  cavalry  squadron.  The  primary  training  audience  was  augmented  by  an 
extended  training  audience  (21  members),  which  included  three  maneuver  company  commanders 
and  their  deputies,  three  maneuver  platoon  leaders  and  one  mortar  platoon  leader  for  one  of  the 
companies,  a  battery  commander  and  his  deputy,  a  forward  support  company  commander  with 
his  deputy,  and  the  scout  platoon  leader  and  platoon  sergeant.  A  total  of  35  soldiers  from  the 
squadron  participated  in  the  BCR  HI  as  members  of  the  primary  training  audience  and  extended 
training  audience. 

A  team  of  RAs  was  formed  to  conduct  the  system  training  and  assist  during  the  BCR  HI 
trials.  The  RAs  began  their  preparation  two  weeks  prior  to  the  train-up  week  to  leam  how  to 
operate  the  SC'*  system.  Four  of  the  RAs  provided  technical  assistance  and  system 
troubleshooting  at  the  nodes  during  the  trials,  while  the  others  served  as  workstation  operators 
for  the  company-  and  platoon-level  and  OPFOR  workstations. 

A  White  Cell  was  also  formed  to  provide  brigade-level  context-based  guidance  for  the 
squadron.  The  White  Cell  included  both  MMBL  personnel  and  active  duty  military  personnel  in 
the  roles  of  Battle  Force  Commander,  Intelligence  Officer,  Operations  Officer,  Fire  Support 
Officer,  and  Logistics  Officer. 

Commander  and  Staff  Organization 

One  major  feature  for  the  BCR  HI  was  the  reengineered  battalion  command  group 
structure  for  the  primary  training  audience,  depicted  in  Figure  4.  It  comprised  four  staff  nodes: 
the  command  group  node  (Command  1),  the  deputy  commander  node  (Command  2),  the  current 
operations  node  (Control  1),  and  the  future  operations  node  (Control  2).  Each  node  was  staffed 
by  3  or  4  primary  staff  members. 

This  structure  was  similar  to  the  structure  settled  on  by  the  BCR  n  battalion  commander. 
The  BCR  III  training  audience  began  with  this  structure,  and  was  given  some  information  about 
the  roles  and  responsibilities  of  each  staff  member  in  each  node.  As  the  BCR  HI  progressed,  the 
battalion  commander  was  allowed  to  realign  roles  and  responsibilities  to  try  out  different 
structures. 
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Figure  4,  Reengineered  battalion  command  group  structure. 

BCR  in  Timeline 

During  the  three-week  period,  the  events  included  5  days  of  participant  training,  a  2-day 
pilot  test,  and  the  7-day  trial  (see  Figure  5).  The  participant  training  included  ModSAF 
familiarization  as  well  as  training  on  staff  processes  and  decision-making  (primarily  to  ensure  a 
common  working  model  of  the  processes)  with  special  attention  to  the  staff  processes  in  digital 
environments.  This  train-up  concluded  with  exercises  in  mission  planning  and  execution  to 
reinforce  the  ModSAF  training  and  allow  time  and  input  for  the  commander  to  refine  the  staff 
processes.  The  pilot  test  consisted  of  a  “rehearsal”  of  the  trials  and  data  collection  procedures. 
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Figure  5.  BCR  m  schedule. 

The  trials  involved  the  squadron  in  planning,  preparation,  and  execution  of  six  missions, 
equally  divided  between  offensive  and  defensive  operations.  The  squadron  was  issued  a  brigade 
operations  order  (OPORD)  which  established  tactical  tasks  for  the  unit  and  set  the  conditions 
under  which  it  would  operate.  Subsequent  FRAGOs  modified  the  tasks  and  conditions  as  the 
BCR  in  progressed  from  one  mission  to  another.  Feedback  sessions  (referred  to  as  hotwashes) 
were  conducted  after  each  mission  to  identify  software  or  hardware  performance  shortfalls,  to 
elicit  participant  recommendations  for  improving  the  system,  and  to  explore  the  objective  SC^ 
system’s  impact  on  future  DTLOMS. 
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Changes  During  Pilot  Implementation 


The  most  significant  changes  from  the  prototype  future  staff  training  package  that 

occurred  during  the  pilot  implementation  included: 

•  Unit  preparation:  Since  the  unit  did  not  have  detailed  preparatory  materials  before  coming  to 
the  BCR  in,  they  worked  on  development  of  staff  processes  and  techniques  that  were  not 
based  on  BCR  I  and  H  discoveries,  and  that  were  not  entirely  in  line  with  the  assumptions 
underlying  the  Level  2-4  training. 

•  Entry  level  skills:  Another  assumption  for  the  training  was  that  the  staff  members  would  be 
familiar  with  their  duties  and  responsibilities  in  a  conventional  staff  configuration.  Due  to 
normal  turbulence  within  the  unit,  however,  many  of  the  staff  members  were  relative  novices 
in  staff  operations.  This  meant  that  some  of  the  training  was  too  intensive,  attempting  to 
build  on  skills  that  the  individuals  were  lacking.  As  a  result,  the  time  allotted  for  Levels  2-4 
was  extended,  leaving  less  time  than  planned  for  the  TDXs. 

•  SC*  equipment  use:  The  BCR  HI  plans  called  for  use  of  helmet-mounted  displays  (HMDs) 
for  all  members  of  the  Control  2  node,  rather  than  the  normal  computer  screen  displays  found 
in  the  other  nodes.  The  training  was  structured  to  provide  separate  training  for  personnel  in 
that  node,  so  that  they  could  use  the  HMDs  at  all  times.  This  compromised  the  Level  3 
training,  where  participants  were  to  be  grouped  for  training  according  to  their  node  roles, 
rather  than  being  grouped  within  their  nodes.  Even  before  the  end  of  the  training  week,  the 
MMBL  determined  that  the  HMDs  were  not  effective,  and  in  fact  caused  equipment  failures. 
As  a  result,  participants  were  switched  to  the  normal  display  modes,  but  by  that  time  they  had 
already  begun  to  participate  in  Level  4  training. 

•  Training  schedule:  The  plans  for  the  train-up  in  the  program  of  instruction  (POI)  were  within 
the  allotted  time  of  five  ^ys,  but  there  was  very  little  room  for  adjustment.  Several 
additional  activities  (e.g.,  satellite  training)  were  inserted  into  the  training  week,  which 
extended  the  training  day  for  some  soldiers. 

•  Training  session  direction:  Early  design  and  development  of  the  Level  4  and  Level  5  staff 
process  training  had  proceeded  on  the  assumption  that  the  sessions  would  be  facilitated  by 
members  of  the  project  team.  Later  it  was  determined  that  the  Level  5  sessions  would  be 
initially  conducted  by  the  MMBL  Training  Director  and  later  transitioned  to  the  Squadron 
Commander.  Because  the  Training  Director’s  time  was  limited  during  the  week  prior  to  the 
BCR  HI,  efforts  to  prepare  him  for  that  role  were  similarly  limited.  Since  time  was  Ihmted 
during  the  training  week  to  brief  the  Squadron  Commander  on  the  design  of  the  team  training 
sessions,  he  was  unable  to  fully  implement  the  requirements. 

•  Performance  feedback:  There  were  three  related  issues  that  completely  compromised  the 
performance  feedback  plans.  First,  the  evaluation  plan  was  based  on  the  assumption  that 
survey  data  would  be  collected  by  means  of  computer-based  software  systems  that  would 
allow  very  quick  summaries  of  results  that  could  be  used  for  feedback.  Second,  the 
automated  data  collection,  using  the  DCA,  was  assumed  to  have  been  quickly  available  for 
feedback.  Third,  team  training  sessions  that  were  conducted  during  the  training  week  were  to 
have  continued  into  the  trials.  However,  use  of  the  survey  software  would  have  led  to  an 
increased  risk  for  data  security  as  well  as  increased  data  junction  loads,  so  surveys  were 
administered  in  a  paper-and-pencil  mode  and  could  not  be  summarized  for  feedback  quickly. 
The  DCA  routines  for  automated  data  collection  were  not  available  prior  to  the  BCR  HI,  but 
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rather  were  deferred  to  post-BCR  processing  of  the  data.  Finally,  no  team  training  sessions 
were  scheduled  during  the  trials. 

The  bottom  line  for  these  changes  was  that  some  results  of  the  formative  evaluation  did 
not  address  the  prototype  training  package  design,  and  therefore  data  for  performance  feedback 
purposes  were  either  not  available  or  not  used. 

Formative  Results 

As  discussed  earlier,  data  was  collected  during  BCR  HI  for  two  purposes.  The  first 
purpose  was  to  gain  insight  into  how  the  prototype  staff  training  package  could  be  improved. 
The  second  purpose  was  to  gain  experience  in  developingperformance evaluation  methods  for 
future  brigade  and  battalion-level  staffs  using  advanced  systems  and  operating  in 
geographically  dispersed  locations  on  the  battlefield.  Results  from  both  are  presented  in  this 
section. 


Prototype  Staff  Training  Evaluation 

To  evaluate  the  utility  of  the  prototype  training  package  and  to  document  lessons  learned 
for  future  training  development,  data  were  collected  from  the  training  participants  by  means  of 
surveys,  interviews,  and  observations.  Training  evaluation  surveys  were  administered  seven 
times;  at  the  conclusion  of  Level  2,  Level  3,  Level  4,  TDX  1,  TDX  2-3,  TDX  4,  and  end  of  BCR 
TTT  Interviews  were  conducted  at  the  conclusion  of  training  and  at  the  end  of  the  BCR  HI. 
Observations  were  documented  continuously  throughout  the  training. 

For  this  project,  the  data  collected  concerning  the  Level  5  training  of  TDXs  were  of 
particular  importance  and  interest.  However,  the  training  that  was  recommended  for  the  RA 
preparation  and  for  the  earliCT  part  of  the  train-up  week  (Levels  1, 2, 3,  and  4)  was  also  of  some 
interest.  While  the  project  team  had  not  been  responsible  for  the  RA  training  nor  for  the 
completed  Levels  1-3  training,  and  had  only  partial  responsibility  for  Level  4,  a  sizable  body  of 
data  was  obtained  that  could  inform  revisions  to  the  training  for  future  BCRs  and  for  other  future 
staff  training  for  the  Army.  Therefore,  results  pertaining  to  RA  training  and  Levels  1-4  will  be 
presented  first,  followed  by  more  in-depth  reporting  on  the  Level  5  TDXs  and  Team  Training 
Sessions.  The  Training  and  Measurement  Support  Package  Volume  5  (1999)  presents  the  survey 
comments  and  summaries  of  the  interviews.  Examples  of  the  survey  and  interview  data  can  be 
seen  in  Appendix  F. 

Evaluation  of  Training  for  Research  Assistants 

Information  on  the  RA  preparation  was  obtained  primarily  from  interviews  with  the  RAs 
themselves,  but  is  also  support^  by  comments  on  the  surveys  from  the  training  audience 
members.  The  RA  training  was  planned  for  the  two  weeks  prior  to  the  start  of  the  BCR  HI.  It 
was  planned  that  they  would  receive  SC*  system  training  the  first  week  and  have  the  second 
week  available  for  “practice  teaching”  (with  an  audience)  and  additional  individual  practice. 
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The  comments  received  from  the  exercise  participants,  as  well  as  the  RAs,  generally 
centered  on  the  need  for  the  RAs  to  be  better  prepared.  Though  several  people  indicated  the  RAs 
were  familiar  with  the  SC*  system,  RAs  felt  they  lacked  the  depth  of  knowledge  necessary  to 
fully  train  and  support  the  unit.  The  RAs  indicated  they  felt  that  their  training  was  somewhat 
unstructured.  They  indicated  they  would  have  benefited  by  having  a  “full-time  instructor” 
available  that  first  week  to  thoroughly  train  them  on  the  system  tools.  The  RAs  stated  that  they 
did  not  realize  that  the  training  plan  outlines  provided  to  them  were  developed  for  them  to  use  as 
a  guide  to  train  the  unit  members.  The  RA  training  coordinator  commented  that  their  training 
could  have  been  more  structured  and  better  organized. 

Another  concern  noted  by  both  the  RAs  and  the  RA  training  coordinator  was  that  the 
second  week  plan  for  practice  teaching  never  came  to  fruition.  For  the  most  part,  the  individuals 
slotted  to  be  their  training  audience  were  unable  to  attend  as  planned.  The  RAs  stated  they  had 
at  best  a  few  hours  the  second  week  with  someone  there  to  receive  traimng.  They  felt  the  lack  of 
practice  teaching  hampered  their  ability  to  present  the  information  as  it  was  structured. 

In  general,  the  RAs  and  the  training  coordinator  found  the  training  plan  outlines  to  be 
correct  and  potentially  useful,  both  for  their  own  preparation  (supplementing  formal  instruction) 
and  as  a  checklist  to  use  in  training  the  participants.  The  RAs  pointed  out  that  some  sort  of 
checklist  for  them  to  use  would  have  helped  to  structure  the  training  more  effectively. 

Levels  1-4:  SC^  Training 

Although  the  surveys  and  interviews  did  not  specifically  address  the  Level  1  training 
(initial  orientation),  participants  did  provide  some  comments,  both  on  the  surveys  and  in 
interviews,  on  how  it  could  be  improved.  The  most  frequently  voiced  comment  was  that  the 
orientation  should  be  reduced  or  dropped  in  favor  of  more  practice  or  hands-on  training  time. 
Several  participants  reported  that  the  on-site  briefing  was  a  repeat  of  what  was  presented  at  then- 
home  station,  while  others  commented  that  it  seemed  to  be  addressed  to  upper  echelon  leadership 
rather  than  to  the  participants  themselves. 

The  on-site  briefing  was  not  structured  as  a  Level  1  advance  organizer.  As  a  result,  some 
participants  indicated  that  they  did  not  know  how  the  training  was  organized  and  what  they  were 
expected  to  learn;  the  RAs  also  indicated  that  the  orientation  did  not  clearly  present  the  training 
topics  and  how  the  training  would  be  conducted.  Participants  suggested  changing  the  names  of 
the  levels  of  training  to  something  more  meaningful,  or  explaining  the  levels  more  clearly.  They 
also  suggested  additional  topics,  such  as  supply  and  logistics  information,  enemy  capabilities, 
and  BLUFOR  system  capabilities.  This  information  would  have  helped  the  participants 
considerably  if  it  were  provided  during  pre-BCR  preparation  or  even  in  the  initial  on-site 
orientation.  The  RAs  felt  that  information  about  operating  the  system  itself  should  also  have 
been  included  in  the  unit’s  preparation  package,  although  the  utility  of  such  information  would 
likely  be  limited  because  there  would  be  no  systems  on  which  to  practice. 
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The  data  collected  from  the  training  surveys  for  Levels  2-4  included  ratings 
(Disagree/Agree)  of  statements  as  well  as  written  comments  on  the  statements.  All  statements 
were  rated  on  a  5-point  scale,  ranging  from  “Strongly  Disagree”  to  “Strongly  Agree.”  Table  8 
presents  the  means  for  each  rated  item.  The  items  asked  for  a  rating  on  several  topics  concerning 
training,  including  whether; 

1.  the  day’s  training  sequence  was  logical; 

2.  there  was  enough  formal  instruction  during  the  training; 

3.  there  was  enough  coaching  during  the  training; 

4.  there  was  enough  time  to  practice  during  the  training;  and 

5.  prior  level(s)  of  training  prepared  them  for  the  current  level  of  training. 

For  all  statement  ratings,  a  “1”  signified  that  the  participant  strongly  disagreed  with  the 
statement,  a  “3”  meant  neither  agree  nor  disagree,  and  a  “5”  signified  that  the  participant 
strongly  agreed  with  the  statement.  Standard  deviations  are  shown  in  parentheses. 

Table  8 

Mean  Ratings  on  Levels  1-4  Training 


Item  Content 

Training 

Phase 

Logical 

Sequence 

Enough 

Instruction 

Enough 

Coaching 

Enough  Time 
to  Practice  on 
SC^  Systems 

Prior  Training 
was  Helpful 

Level  2 

3.74 

(0.74) 

3.74 

(0.66) 

3.80 

(0.72) 

3.23 

(1.24) 

— 

Level  3 

3.54 

3.74 

3.89 

4.09 

3.74 

(0.61) 

(0.56) 

(0.68) 

(0.66) 

(0.74) 

Level  4 

3.60 

3.43 

3.63 

3.51 

3.66 

(0.69) 

(0.88) 

(0.73) 

(0.95) 

(0.76) 

Note.  Scale  values  were  l=Strongly  Disagree;  2=Disagree:  3=Neither  Disagree  nor  Agree;  4=Agree;  5=Strongly 
Agree.  Dashes  (— )  indicate  the  question  did  not  appear  on  that  survey. 


Overall,  average  ratings  ranged  from  3.23  to  4.09.  For  each  statement  and  for  each 
Level,  the  majority  of  participants  gave  a  response  of  “4”  or  “5”  (Agree  or  Strongly  Agree);  the 
modal  response  for  each  of  the  above  statements  for  each  Level  of  training  was  “4.”  This 
indicates  Aat  overall,  the  participants  found  the  training  to  be  adequate  in  terms  of  sequencing 
and  amount  of  instraction,  coaching,  and  practice. 

However,  within  the  overall  positive  reaction  to  the  training,  some  items  were  somewhat 
less  positively  rated.  On  the  item  concerning  the  amount  of  practice  time  at  Level  2, 35%  of 
respondents  (43%  of  primary  training  audience  respondents)  were  somewhat  negative,  and  their 
comments  support  the  ratings.  Several  of  them  suggested  more  exercises  and  more  hands-on 
time  with  the  equipment  as  ways  to  improve  the  training,  and  two  indicated  that  the  pace  of  the 
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training  was  too  fast,  not  allowing  enough  time  to  review  and  digest  the  information.  In 
conversation  with  the  participants,  a  number  of  them  pointed  out  that  they  did  not  have  a  clear 
understanding  of  the  structure  of  the  train-up  and  no  definition  of  the  training  objectives. 

Comments  on  the  Level  3  training  again  included  the  lack  of  explicitly-stated  objectives, 
although  respondents  reported  that  they  were  beginning  to  understand  the  stmcture  or  flow  of  the 
training.  They  also  expressed  the  need  for  more  practice  time,  and  the  extended  training 
audience  indicated  that  they  should  have  had  more  formal  instruction  at  Level  3.  Four 
respondents  commented  on  the  training  pace:  two  indicated  that  the  training  was  too  fast,  one 
felt  it  was  too  slow,  and  one  reconunended  having  a  self-paced  tutorial.  However,  they  did  begin 
to  recognize  the  training  progression,  from  individual  to  small  group  and  from  simple  to  complex 
(information  which  was  included  but  not  presented  in  the  orientation).  Two  respondents 
suggested  that  the  training  should  include  structured  practice  exercises  or  some  standards  to 
evaluate  whether  the  material  had  been  learned.  It  should  be  noted  that  the  training  plan  outlines 
did  contain  recommendations  for  structured  practice  exercises. 

The  training  design  for  Level  3  called  for  members  of  the  primary  training  audience  to  be 
clustered  according  to  their  jobs,  so  that  those  with  similar  functions  would  be  trained  together 
and  they  could  begin  to  develop  inter-node  tactics,  techniques,  and  procedures  (TI  Ps)  within 
functions.  Respondent  comments  on  this  plan  were  mixed.  Some  recognized  that  the  clustering 
provided  opportunities  for  sharing  ideas  and  techniques.  However,  at  least  one  RA  noted  that  it 
interfered  with  the  learning  on  the  system  itself.  The  implication  is  that  Levels  2  and  3  training 
should  be  more  thorough  before  Levels  4  and  5  training  begins. 

There  were  several  comments  on  the  surveys  and  in  the  interviews  concerning  the 
underlying  structure  for  the  Level  3  training.  As  designed,  the  training  contained  partial 
scenarios  with  moving  enemy  units,  fuel  and  ammunition  consumption,  and  unit  status  r^rts. 
These  scenarios  provided  the  situation  for  showing  various  system  ftmctions.  However,  several 
respondents  commented  that  they  should  have  had  such  scenarios,  apparently  unaware  that  the 
information  was  already  included.  This  speaks  more  to  the  implementation  of  training  than  to  its 
design. 


Level  4  training  was  their  first  opportunity  to  practice  tasks  in  the  context  of  a  tactical 
scenario.  As  shown  in  Table  8,  the  responses  were  positive,  but  indicate  room  for  improvement; 
the  mode  for  all  items  was  “4.”  The  exception,  at  Level  4,  was  on  the  item  concerning  the  utility 
of  the  prior  training:  36%  of  the  primary  training  audience  indicated  that  the  prior  training  was 
not  helpful.  Their  conunents  covered  a  range  of  suggested  improvements,  including 
reorganizing  the  flow  and  sequence  of  topics  like  conferencing,  providing  for  a  more  direct 
multi-echelon  linkage  during  the  exercises  (e.g.,  having  companies  and  platoons  working  more 
in  concert  with  the  squadron  staff),  giving  more  precise  definitions  of  roles  and  functions  for 
each  node  and  position,  incorporating  more  brigade-level  interaction,  and  providing  more 
practice  on  the  UAVs  and  scout  vehicles.  Three  of  the  comments  indicated  that  the  participants 
were  beginning  to  develop  TTPs  pertinent  to  the  plaiming  process. 

At  the  conclusion  of  the  BCR  HL  the  participants  were  asked  again  for  their  suggestions 
on  the  training,  by  means  of  surveys  and  interviews.  The  two  primary  comments  during  the 
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training  were  repeated;  that  the  training  objectives  should  be  more  clearly  stated,  and  that  the 
participants  wanted  more  practice  time.  Their  recommendations  on  the  topics  to  be  included  in 
the  training  were  focused  on  the  SC^  tools  and  on  system  capabilities.  They  requested  (again) 
practice  exercises,  and  two  of  the  respondents  suggested  that  the  training  be  delivered  by  means 
of  a  computer-based  tutorial.  Several  of  the  RAs  and  observer/controllers  (O/Cs)  also  suggested 
using  tutorials  for  individual  training.  While  such  an  approach  would  allow  each  participant  to 
receive  standardized  and  structured  instruction  at  an  appropriate  pace,  it  would  not  obviate  the 
necessity  for  small  group  practice  and  collective  training  exercises.  Small  group  and  collective 
tutorials  would  require  significant  design  and  development  work. 

Level  5:  Tactical  Decision-Making  Exercises 

The  data  collected  from  the  training  surveys  for  the  Level  5  TDXs  included  ratings 
(Disagree/Agree)  of  statements  as  well  as  written  comments  on  the  statements.  Table  9  presents 
the  means  for  each  rating  item.  The  items  asked  for  a  rating  on  several  topics,  including 
whether: 

1.  the  prior  training  (individual  and  small  group)  prepared  them  for  the  TDX; 

2.  the  TDX  gave  them  the  chance  to  practice  using  the  SC^  system; 

3.  the  tactical  materials  were  sufficient;  and 

4.  prior  TDXs  prepared  them  for  the  current  TDX. 

Table  9 

Mean  Ratings  on  Level  5  TDXs  Training 


Phase 

hem  Content 

Prior  Training  was 
Helpful 

Enough  Time  to 
Practice  on  SC* 
Systems 

Tactical  Materials 
Were  Sufficient 

Prior  TDX(s)  Were 
Helpful 

3.91* 

3.66* 

3.55" 

TDX  1 

(0.37) 

(0.97) 

(0.71) 

3.77* 

3.89* 

3.58' 

3.71* 

TDX2&3 

(0.65) 

(0.87) 

(0.66) 

(0.62) 

3.94* 

3.85” 

3.47” 

3.83* 

TDX  4 

(0.48) 

(0.74) 

(0.89) 

(0.51) 

Note.  Scale  values  were  l=Strongly  Disagree;  2=Disagree;  3=Neither  Disagree  nor  Agree;  4=Agree;  5=Strongly 
Agree.  *N=35;  '^=34;  ‘N=33.  Dashes  (-)  indicate  the  question  did  not  appear  on  that  survey. 


For  all  statement  ratings,  a  “1”  signified  that  the  participant  strongly  disagreed  with  the 
statement,  a  “3”  meant  neither  agree  nor  disagree,  and  a  “5”  signified  that  the  participant 
strongly  agreed  with  the  statement.  Standard  deviations  are  shown  in  parentheses.  Once  again, 
the  mean  ratings  show  that  the  general  response  was  positive,  and  the  modal  response  for  every 
item  was  “4”  (Agree). 


52 


On  the  first  TDX,  many  comments  again  centered  on  the  need  for  practice  and  repetition. 
Respondents  also  suggested  the  need  for  more  training  on  specific  functions,  such  as  radio 
operations,  flying  the  UAVs,  creating  and  using  graphics,  tracking  sensors,  and  driving  the 
reconfigurable  vehicles.  Several  of  the  primary  training  audience  members  felt  that  the  pace  of 
the  training  could  have  been  a  little  more  intense;  the  extended  training  audience  members  were 
more  critical,  stating  that  all  they  did  was  drive  or  move. 

The  focus  for  many  comments  on  TDX  1  shifted  from  system  issues  to  more  tactical  and 
staff  process  issues.  The  respondents  indicated  that  they  were  in  need  of  or  were  developing 
TTPs  for  various  activities,  including  driver  tasks,  fire  support  element  (FSE)  techniques,  and 
working  with  subordinate  units.  One  participant  requested  TTPs  from  prior  BCRs. 

Comments  on  the  scenario  and  tactical  materials  were  specific  and  constractive.  Two 
respondents  were  critical  of  the  lack  of  logistics  information  and  logistics  requirements.  Several 
comments  concerned  the  perceived  incompleteness  of  the  brigade-level  orders  and  graphics.  For 
the  most  part,  however,  they  indicated  that  the  tactical  cues  for  the  TDX  provided  an  adequate 
set  of  conditions  for  performance. 

The  TDXs  2  and  3  were  conducted  as  a  continuous  activity,  and  therefore,  separate 
comments  on  the  two  TDXs  were  not  obtained.  Responses  on  TDXs  2  and  3  were  positive  on 
the  utility  of  the  prior  training,  although  they  mentioned  specific  topics  on  which  they  wanted 
more  training  or  information  (logistics,  training  with  subordinate  units,  non-line  of  sight 
[NLOS],  overlays,  and  UAVs).  Responses  indicated  that  the  training  flow,  from  basic  to  more 
complex  activities,  was  appropriate.  They  also  felt  that  the  prior  TDX  had  provided  valuable 
practice,  although  the  importance  of  additional  practice  continued  to  be  a  focus.  Several 
comments  again  indicated  that  the  squadron  was  developing  TTPs  for  planning,  use  of  graphics, 
and  use  of  the  whiteboard  conferencing. 

The  TDX  4  was  the  most  intensive  of  the  train-up  phases,  and  respondents  mentioned 
that  it  was  painful  but  necessary.  The  insufficiency  of  CSS  play  was  mentioned  again  as  a 
drawback  within  the  training.  They  continued  to  work  out  their  TTPs  concerning  sending 
OPORDs,  use  of  FRAGOs  and  warning  orders  (WARNOs),  labeling  tactical  products  that  are 
saved  on  the  system,  and  prioritization  and  dissemination  of  information. 

In  summary,  participants  focused  their  comments  on  the  need  for  practice  opportunities 
(at  the  expense  of  briefings  and  lectures,  if  necessary),  explicitly-stated  training  objectives,  and 
integrated  training  across  echelons. 

Team  Training  Sessions 

In  addition  to  evaluating  training  on  the  SC^  systems,  the  participants  evaluated  the  Team 
Training  Sessions,  including:  the  information  management  session  (included  in  the  Level  4 
training),  a  roles  and  functions  session  (included  in  the  Level  4  training),  the  Pre-Action 
Analysis,  Commander’s  Timeout,  and  Team  Decision-Making  Debrief  (all  intended  to  be 
conducted  throughout  Level  5,  in  the  TDXs). 
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In  general,  the  evaluation  data  and  comments  from  the  training  audience  have  limited 
validity  and  applicability  for  a  number  of  reasons.  The  information  management  and  roles  and 
functions  sessions  were  not  evaluated  explicitly  in  the  survey  questions  and  therefore  could  only 
be  evaluated  if  individuals  singled  them  out  for  comments  in  open-ended  questions.  The  Pre- 
Action  Analysis,  Commander’s  Timeout,  and  Team  Decision-Making  Debrief  were  addressed 
explicitly  in  the  survey.  However,  they  were  not  used  consistently  during  the  TDXs  for  a  variety 
of  reasons,  including  exercise  time  constraints  and  lack  of  discrete  start  and  end  points  for  the 
TDXs,  as  conducted.  Additionally,  several  participants  and  0/Cs  pointed  out  that  the  Squadron 
Commander  already  had  a  method  for  sharing  his  situational  awareness  and  intent,  and  for 
preparing  and  debriefing  the  staff. 

The  discussion  below  is  based  on  the  data  and  comments  that  were  related  to  the  Team 
Training  Sessions  and  should  be  examined  within  the  limitations  outlined  above.  These  data 
were  collected  at  the  end  of  TDX  1,  at  the  end  of  TDX  3  (encompassing  both  TDX  2  and 
TDX  3),  at  the  end  of  TDX  4  (the  end  of  the  training  week),  and  at  the  end  of  the  BCR  HI  (Day 
15).  Training  evaluations  from  the  participants  and  especially  the  Squadron  Commander  are  ^so 
mentioned  where  appropriate  and  informative. 

Information  management.  The  only  data  that  were  collected  about  the  information 
management  session  were  in  the  form  of  comments  on  the  surveys  and  interviews.  Comments 
from  several  participants  suggested  that  the  information  management  session  was  not  the  best 
use  of  their  time  during  the  train-up  week,  because  the  material  presented  had  been  covered  in 
other  training  or  was  just  a  part  of  their  everyday  way  of  doing  business.  However,  conunents 
following  the  later  TDXs  reflected  that  the  unit  was  still  dealing  with  information  management 
issues  (“too  much  information,”  “information  overload,”  “still  need  to  work  through 
prioritization  and  dissemination”).  One  node  commander  indicated  that  his  difficulty  knowing 
what  the  commander  wanted  at  times  was  normal  (although  it  should  be  noted  that  “normal”  is 
not  the  same  as  “inevitable”).  These  comments  reflect  a  need  for  information  management 
training,  and  suggest  that  the  training  should  address  the  issues  that  arise  with  Battle  Command 
using  C^I  systems  (for  example,  at  the  level  of  file  management,  overlay  management, 
dissemination,  and  manipulation).  Other  comments  reflected  the  perception  that  the  information 
management  session  was  aimed  solely  at  the  primary  training  audience  and  ignored  the  extended 
training  audience. 

On  reflection,  this  session  appeared  at  the  wrong  time  in  the  training.  It  was  presented 
when  most  of  the  trainees  were  still  thinking  about  individual  procedural  skills,  before  the  TDXs. 
Only  after  they  began  work  as  a  multi-node  staff  did  the  information  management  problems 
surface.  Presenting  the  training  prior  to  their  having  any  experience  with  the  difficulties  of 
dealing  with  a  heavy  information  load  did  not  allow  them  to  relate  the  training  content  to 
personal  experience.  Although  some  of  the  negative  comments  about  this  session  came  from  the 
more  senior  officers,  others  in  the  unit  found  the  information  management  issues  to  be 
challenging  in  the  TDXs.  This  session  may  be  more  useful  when  some  information  management 
tasks  are  about  to  be  performed,  by  addressing  file  management  and  management  of  transferred 
whiteboards,  overlays  and  files. 
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Another  shortcoming  in  this  training  was  the  use  of  e-mail  as  the  basis  for  a  short  group 
exercise  for  the  primary  training  audience,  since  e-mail  was  rarely  used  in  the  BCR  environment. 
Most  communication  was  conducted  over  voice/radio  and  via  the  downloading  and  sending  of 
files  through  the  network.  These  examples  would  have  been  more  relevant  for  the  training, 
especially  if  the  training  occurred  later  in  the  week. 

Roles  and  functions.  The  roles  and  functions  training  session  was  not  evaluated 
explicitly  in  the  surveys;  however,  several  training  survey  comments  suggest  that  some  people 
were  not  really  sure  what  tasks  they  were  supposed  to  be  doing  during  the  training  (e.g.,  “I  am 
finding  it  difficult  to  find  my  niche  as  effects  guy. . .  what  do  I  do?”  and,  “streamline  who 
delegates  to  me. , Although  unit  personnel  began  to  have  questions  about  their  role  early  in 
the  training,  the  scheduling  of  the  roles  and  functions  session  which  called  for  the  session  to  be 
conducted  on  the  third  day  of  training  was  appropriate  since  most  soldiers  would  not  have  been 
exposed  to  enough  of  the  capabilities  of  the  system  prior  to  this  point  to  understand  what 
they  were  supposed  to  be  doing  or  how  their  role  or  function  could  be  modified  to  better  support 
the  commander. 

Pre-action  analysis,  commander’s  timeout,  and  team  decision-making  debrief.  A  Pre- 
Action  Analysis  was  conducted  at  the  beginning  of  TDX  1;  however,  it  was  not  folly 
implemented  as  designed.  Subsequently,  the  Commander  and  Traimng  Director  chose  not  to 
conduct  formal  Pre- Action  Analyses,  although  they  did  spend  a  few  minutes  at  the  start  of  each 
day  discussing  the  tactical  scenario  for  the  upcoming  TDX;  thus  the  evaluations  reflect  a  limited 
implementation  of  this  technique.  In  addition,  the  Commander’s  Timeout  was  never  utilized  in 
the  intended  manner,  where  the  Squadron  Commander  would  give  his  situational  awareness  and 
elicit  reactions  and  other  viewpoints  from  the  rest  of  the  staff  and  company  commanders. 

Instead,  the  Squadron  Commander  either  called  a  huddle  for  his  node  OICs  or  his  company 
commanders,  but  did  not  call  a  time-out  for  all  the  staff  and  company  commanders  nor  ask  for 
others’  inteipretations  of  the  situation. 

Finally,  the  Team  Decision  Making  Debrief  was  not  implemented  as  intended.  Due  to 
the  TDXs  having  no  clear  start  and  end  events,  as  conducted,  it  was  difficult  to  use  distinct  TDX 
boundaries  as  the  cut-offs  for  discussing  certain  issues,  and  as  a  result,  the  debriefs  focused  on 
the  day’s  events  rather  than  specific  TDX  objectives.  In  general,  these  sessions  reviewed 
processes  to  sustain  and  improve  unit  tactical  performance  and  discussion  of  TTPs  rather  than 
decision-making  processes,  workload  distribution,  or  situational  awareness. 

Table  10  presents  the  means  for  each  instance  participants  rated  these  three  sessions.  The 
sessions  were  each  rated  on  whether  they  were  helpful  and  would  be  useful  in  future  training 
(end  of  training  and  end  of  BCR  HI  only).  For  all  statement  ratings,  a  “1”  sigmfied  that  the 
participant  strongly  disagreed  with  the  statement,  a  “3”  meant  neither  agree  nor  disagree,  and  a 
“5”  signified  that  the  participant  strongly  agreed  with  the  statement  Standard  deviations  are 
presented  in  parentheses. 
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Table  10 


Team  Training  Session  Means 


Item 

Team  Decision- 

Pre-Action  Analvsis 

Commander’s  Timeout 

Making  Debrief 

Phase 

Helpful  Future 

Helpful  Future 

Helpful  Future 

TDX  1 

3.51* 

(0.78) 

— 

3.29' 

(0.82) 

— 

3.50” 

(0.79) 

TDX  2  &  3 

3.56” 

(0.66) 

— 

3.35' 

(0.61) 

- 

3.61' 

(0.97) 

— 

TDX  4 

3.57* 

(0.65) 

3.77* 

(0.73) 

3.35* 

(0.71) 

3.48* 

(0.77) 

3.76' 

(0.66) 

3.82' 

(0.73) 

End  of  Training 

3.69" 

(0.97) 

~ 

3.41" 

(0.87) 

— 

3.69" 

(1.03) 

Note.  Scale  values  were  l=Strongly  Disagree;  2=:Disagree;  3=Neither  Disagree  nor  Agree;  4= Agree;  5=Strongly 
Agree.  *N=35; *^<=34;  *14=33;  ‘*N=32;  'N=3 1 .  Dashes  (-)  indicate  the  question  did  not  appear  on  that  survey. 


The  modal  response  for  all  Pre- Action  Analysis  ratings  was  “4,”  which  signifies  that  in 
general,  respondents  appeared  to  believe  that  this  technique  was  helpful  during  the  TDXs  and 
would  be  something  that  they  would  find  useful  in  future  training.  The  extended  training 
audience  (company-level  and  below)  found  the  Pre- Action  Analysis  sessions  less  useful  than  did 
the  primary  training  audience.  One  respondent  suggested  that  the  session  should  be  used  to  set 
critical  objectives  and  focus  problem-sOlvihg  (as  had  been  intended  when  the  session  was 
designed).  In  one  interview,  a  node  commander  referred  to  the  Pre- Action  Analysis  as  “critical, 
but  not  used  properly  or  effectively.” 

The  modal  response  for  the  Commander’s  Timeout  ratings  was  “3,”  which  signifies  that 
in  general,  respondents  believed  this  technique  was  not  helpful.  Over  the  three  days  of  TDXs, 
the  Commander  had  two  informal  meetings  with  the  node  OICs  and  with  the  company 
commanders  as  an  opportunity  for  these  people  to  “recalibrate.”  These  meetings  did  not  include 
other  members  of  the  staff,  and  so  did  not  serve  as  an  opportunity  for  the  other  staff  inembers  to 
leam  from  the  Commander’s  expertise  in  sizing  up  the  situation.  Because  of  this,  the  extended 
training  audience  in  particular  felt  that  the  Commander’s  Timeout  sessions  were  not  helpful. 
Several  participants  commented  that,  in  general,  stepping  aside  with  the  Commander  and 
reviewing  his  intent  and  his  situational  awareness  is  a  vital  activity  for  the  staff.  In  interviews, 
there  was  a  strong  show  of  support  for  the  training  value  of  situational  awareness  timeouts,  as 
required.  In  addition,  when  asked,  the  Commander  stated  that  he  understood  the  importance  of 
his  role  in  shaping  other  people’s  understanding  of  the  situation  by  asWng  questions  and  probing 
for  interpretation  of  events  and  information. 
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The  modal  response  for  all  Team  Decision-Making  Debrief  ratings  was  “4,”  which 
indicates  that  overall,  participants  thought  it  was  useful.  Several  of  the  respondents  commented 
that  the  debrief  sessions  were  really  just  after  action  reviews  (AARs).  Although  the  A\R 
concept  is  appropriate  for  a  focus  on  tactical  issues  and  the  content  of  tactical  decisions,  the 
intent  for  the  debriefs  was  to  focus  the  learning  points  on  the  team  decision  making  process.  A 
focus  on  shared  situational  awareness,  information  management,  workload,  and  roles  and 
funetions  in  relation  to  decision-making  could  have  been  a  valuable  addition  to  the  debrief 
sessions. 

.<\nTnmarv  of  Training  Method  Findings 

The  results  of  the  surveys,  interviews,  and  observations  described  above  provide 
information  concerning  both  the  structure  and  the  content  of  the  training.  Structure  includes  the 
training  flow,  implementation  model,  training  scenarios,  and  support  materials.  Even  though  the 
training  was  not  conducted  as  designed,  it  appears  that  the  progression  from  individual  to  small 
group  to  collective  training  is  appropriate.  Respondents  who  recognized  the  progression 
affirmed  it;  those  who  did  not  recognize  it  requested  such  a  progression.  In  particular,  they 
indicated  that  the  orientation  could  have  been  more  useful  in  preparing  them  for  the  train-up  that 
followed. 

The  training  sequence,  involving  a  very  short  period  of  SC^  system  introduction  (Level 
2),  followed  by  small  group  functional  training  outside  the  node  structure  (Level  3)  and  intra- 
and  inter-node  training  (Level  4),  requires  some  examination.  The  initial  tools  training  did  not 
contain  enough  formal  instruction,  practice,  or  skill  assurance.  As  a  result,  during  small  group 
training,  individuals  were  still  attempting  to  learn  individual  skills.  The  Level  4  training  which 
combined  staff  training  with  company  commander  and  platoon  leader  training  (multi-echelon) 
was  frustrating  for  some  participants  at  both  the  squadron  and  comply  levels. 

The  TDXs  were  seen  as  an  important  component  of  preparation.  While  the  mission 
segments  (planning,  rehearsal,  and  execution)  were  acceptable,  the  scenarios  may  have  been  both 
too  slow-paced  and  too  deliberate.  By  incorporating  more  high-stress  situations,  the  staff  would 
have  had  more  incentive  to  conduct  situational  awareness  recalibrations  and  intensive  decision¬ 
making  debriefs.  Some  participants  also  suggested  more  TDXs  so  they  would  have  had  more 
opportunities  to  practice  and  refine  their  TTPs. 

The  support  materials  for  the  TDXs  (e.g.,  INTSUMs  and  FRAGOs)  were  seen  as 
adequate.  The  single  item  most  conspicuous  by  its  absence  was  communication  of  the  training 
objectives  to  the  participants.  Since  the  overall  training  objectives  were  not  covered  as  intended 
in  the  initial  orientation,  these  objectives  should  have  been  more  clearly  stated  as  the  training 
progressed.  Revisions  to  the  delivery  of  the  training  materials  should  include  this  simple  but 
critical  feature.  Other  suggestions  pertaining  to  materials  focused  on  training  checklists  for  the 
RAs,  job  aids  showing  S^  operations,  charts  of  BLUFOR  and  OPFOR  systems  and  capabilities, 
and  roadmaps  of  objectives  and  topics  for  the  training  week. 

Results  concerning  the  training  content  centered  on  SC^  topics,  battlefield  environment 
information,  and  staff  process  training.  In  general,  parti^ants  found  their  initial  training  to  be 
only  adequate  in  terms  of  what  they  learned  about  the  SCT  system.  Suggested  fixes  ranged  from 
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more  time  for  structured  practice,  to  better-trained  RAs,  to  detailed  job  aids.  In  addition, 
inclusion  of  CSS  functions  was  generally  judged  inadequate. 

Reactions  to  the  team  training  sessions  were  mixed.  Most  of  the  respondents  indicated 
that  this  staff  did  not  need  information  management  training  as  provided,  although  there  was  no 
argument  that  it  was  an  important  topic.  Nonetheless,  they  continued  to  have  information 
management  difficulties  throughout  the  TDXs  and  into  the  trials.  The  timing  of  the  training, 
coming  as  it  did  prior  to  any  significant  information  flow,  may  have  been  the  problem. 

Likewise,  most  participants  agreed  that  the  Pre-Action  Analysis,  Commander’s  Timeout, 
Md  Team  Decision-Making  Debrief  were  valuable  in  concept  but  not  particularly  useful  as 
implemented.  Whether  this  is  due  to  the  way  in  which  they  were  conducted,  or  to  the  way  they 
were  designed  is  unknown.  However,  it  appears  that  the  concepts  should  be  considered  for 
future  use. 


Performance  Evaluation  Methods 

The  performance  data  collection  effort  was  designed  to  gain  experience  with  acquiring 
various  sources  of  information,  especially  digital  information  from  the  SC^  system,  to  enhance 
performance  feedback  to  future  battle  staffs.  The  main  purpose  of  this  section  is  to  examine  how 
well  the  various  evaluation  methods  (automated  measures,  surveys,  interviews  and  observation) 
used  during  the  BCR  m  functioned  so  that  they  can  be  improved  for  future  research  efforts. 
Performance  data  collection  during  the  BCR  HI  was  not  designed  to  evaluate  the  performance  of 
the  unit  since  there  were  no  doctrinal  or  baseline  standards  on  which  to  base  performance 
measures  and  BCR  HI  conditions  did  not  permit  pre-  and  post-training  performance 
measurements  of  the  unit.  However,  sample  results  from  the  performance  data  that  were 
coUected  are  presented  in  this  section.  Complete  BCR  ni  data  summaries  obtained  by  the 
project  team  are  presented  in  Training  and  Measurement  Support  Package  Volume  5  (1999).  In 
addition,  the  project  team’s  analysis  of  the  data  was  provided  to  theMMBL  for  potential 
inclusion  in  the  BCR  DI BLEFR  (1999). 

Sample  Result  -  Assist  in  Decision-Making  Process 

Automated  measures,  surveys,  interviews,  and  observations  were  used  to  obtain  data 
pertaining  to  the  ability  of  the  SC^  system  to  support  the  Commander’s  decision-making  process. 
This  capability  was  amply  demonstrated  through  a  variety  of  automated  measures  that  looked  at 
different  aspects  of  the  SC^  system.  A  major  contributor  to  the  ability  of  the  commander  to  make 
decisions  was  the  performance  of  the  UAVs  which  were  controlled  by  the  squadron,  and  micro- 
UAVs  which  were  controlled  by  the  squadron  scout  platoon.  Over  the  course  of  the  trials,  these 
two  systems  first  detected  51%  of  all  OPFOR  vehicles  in  the  squadron’s  area  of  interest  which 
extended  15  kilometers  beyond  the  squadron’s  area  of  responsibility  (delineated  by  its  rear, 
flank,  and  forward  boundaries  assigned  by  its  higher  headquarters).  In  a  further  breakdown,  the 
two  systems  detected  32%  of  tanks,  53%  of  infantry  fighting  vehicles  (IFV),  and  56%  of  artillery 
vehicles.  When  all  types  of  sensors  were  included,  the  squadron  was  able  to  detect  70%  of 
OPFOR  vehicles.  Interestingly,  the  majority  of  these  vehicles  (64%)  were  first  detected  in  the 
area  of  interest,  beyond  the  area  of  operations,  which  gave  the  squadron  ample  time  to  assess 
OPFOR  strength,  capabilities,  and  intentions  before  they  closed  to  within  range  of  the  squadron’s 
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combat  systems.  For  the  30%  of  OPFOR  vehicles  not  detected,  the  majority  were  second 
echelon  tanks  or  artillery  systems.  None  of  these  systems  had  an  impact  on  the  outcome  of  any 
mission. 

The  common  picture  of  the  tactical  situation  that  the  SCf  system  was  automatically 
providing  to  the  squadron  was  reflected  in  the  type  and  apparently  low  amount  of  squadron 
internal  communications  (see  Figure  6).  Overall,  the  amount  of  time  that  was  spent  by  soldiers 
communicating  with  one  another  using  the  SC^  communications  tools  was  very  low.  The 
amount  of  intercom  usage  is  lower  than  had  been  expected  perhaps  because  the  ambient  noise  in 
the  BCR  in  site  was  low  and  it  was  possible  for  soldiers  within  the  nodes  to  converse  with  one 
another  without  using  the  intercom  system.  The  e-mail  tool  was  rarely  used  during  BCR  HI. 

Another  impact  of  very  good  situational  awareness  was  the  usage  rate  for  various  PVD 
tools.  With  70%  of  the  OPFOR  vehicles  detected  by  the  squadron,  many  PVD  tools  were  used 
minimally  or  not  used  at  all.  Among  all  members  of  the  total  training  audience,  the  overlay 
editor  tool  was  used  by  3 1  soldiers,  the  overlay  file  management  tool  was  used  by  1 1  soldiers, 
and  the  alert  tool  was  used  by  only  7  soldiers.  Additional  research  is  needed  to  determine  why 
the  forward  line  of  own  troops  (PLOT)  display,  field  of  view  (FOV)  display,  plan  tool,  snail 
display,  stealth  control,  and  find  tools  were  used  minimally  or  not  at  all  during  the  trials. 


Figure  6.  Percent  of  time  individual  communication  systems  used. 

The  survey  that  most  directly  addressed  support  of  decision-making  was  the  Operations 
Planning  survey.  Statements  on  the  survey,  which  were  rated  on  a  5-point  scale  (“1”  =  strongly 
disagree,  “5”  =  strongly  agree),  were  divided  into  three  main  factors;  understanding  information 
requirements  (questions  6-10),  understanding  intent  during  mission  planning  (questions  1, 2,  and 
5),  and  understanding  roles  and  functions  (questions  3, 4,  and  1 1).  The  means  for  these  factors 
were  examined  by  day  for  each  node.  As  can  be  seen  in  Figure  7,  the  nodes  rated  themselves 
higher  on  understanding  information  requirements  at  the  end  of  the  BCR  EDI  than  they  did  during 
training.  The  same  thing  occurred  to  a  similar  degree  with  respect  to  understanding  intent  (see 
Figure  8)  and  roles  and  functions  (see  Figure  9). 

Although  the  nodes  report  slight  improvements  in  these  three  areas  by  the  end  of  the  last 
mission,  the  scale  used  may  not  provide  enough  room  for  improvement.  A  7-point  scale  may 
have  shown  more  improvement  from  one  mission  to  the  next.  Another  explanation  is  that  by  the 
end  of  TDX  4,  the  participants  understood  what  their  commander  expected  firom  them.  If  the 
training  provided  them  enough  practice  for  what  they  were  doing  during  the  missions,  it  would 
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Mssion 

Note.  Ratings  based  on  5-point  scale:  1  =  Strongly  disagree;  2  =  Disagree;  3  =  Neither  disagree  nor  agree;  4  = 
Agree;  5  =  Strongly  agree. 

Figure  7.  Information  management  understanding  by  node. 


not  be  expected  that  their  perceptions  of  understanding  would  improve  much  if  they  were  already 
operating  at  a  high  level.  It  may  be  that  an  improved  form  of  this  scale  is  required  to  better 
measure  the  staffs  ability  to  assist  the  Commander’s  decision-making  process  with  the 
reengineered  Battle  Command. 


Agree;  5  =  Strongly  agree. 

Figure  8.  Understanding  intent  by  node. 
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In  the  opinion  of  most  soldier  participants  in  the  exercise,  the  SC**  system  provided  most, 
except  for  CSS-related  statuses,  of  the  information  needed  to  assist  the  commander’s  decision¬ 
making  process.  The  Squadron  Commander  thought  he  had  sufficient  information  except  during 
those  periods  when  UAVs  were  not  available  to  him.  Additionally,  he  thought  that  his  unit  was 
not  fully  exploiting  the  capabilities  of  the  system  since  they  were  still  learning  how  to  use  it. 


—♦—Command  1 
-•-Command  2 
— I — Control  1 
X  Control  2 


TDX4  Pilot  1  Pilot  2  Trial  1  Trial  2  Trial  3  Trial  4  Trial  5  Trial  6 
Mission 


Note.  Ratings  based  on  5-point  scale:  1  =  Strongly  disagree;  2  =  Disagree;  3  =  Neither  disagree  nor  agree;  4  = 
Agree;  5  =  Strongly  agree. 


Figure  9.  Understanding  roles  and  functions  by  node. 

The  observers  confirmed  the  SC^  system  was  providing  almost  100%  situational 
awareness  to  the  commander  and  his  staff.  They  also  noted  that  the  battle  damage  reports  they 
were  receiving  were  very  timely  and  accurate.  By  combining  the  automated  data  measurement, 
survey  results,  interviews,  and  observer  comments,  a  full  picture  of  the  capability  of  the  SC^ 
system  to  support  decision-making  was  developed. 

Sample  Result  -  Team  Performance  and  Processes 


To  study  the  impacts  of  the  reengineered  battle  command  on  team  performance,  survey 
data,  observations,  and  interviews  were  used.  In  the  first  part  of  the  Organizational  Awareness 
survey,  participants  were  asked  to  rate  on  a  5-point  scale  (“1”  =  strongly  disagree,  “5”  =  strongly 
agree)  whether  each  position  m  the  node  was  performing  the  tasks  the  rater  expected  that 
position  to  perform.  Overall,  everyone  felt  that  positions  were  performing  the  tasks  they  were 
expected  to  perform,  with  means  ranging  from  3.98  to  4.14.  In  addition,  participants  were  asked 
to  rate  whether  each  node  was  performing  the  tasks  the  rater  expected  that  node  to  perform. 
Again,  ratings  were  fairly  high  and  consistent,  with  overall  ratings  ranging  from  3.98  to  4.28. 
These  ratings  indicate  that  the  staff  felt  reasonably  confident  about  their  expectations  about  who 
was  doing  what. 
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In  response  to  this  issue,  data  were  also  collected  on  certain  behaviors  by  a  group  of 
observers  using  laptops.  Data  were  collected  on  six  types  of  behaviors;  backup,  communication, 
coordination,  monitoring,  team  orientation,  and  overall.  In  order  to  reduce  bias,  observers 
rotated  nodes  after  every  mission.  The  mean  ratings  can  be  seen  in  Table  1 1. 

The  results  suggest  that  the  command  nodes  were  operating  at  a  high  level  of  team 
performance.  The  control  nodes,  however,  were  mediocre.  This  may  be  due  to  a  lack  of 
workload  in  the  nodes,  failure  to  proactively  do  their  jobs,  or  failure  to  coordinate  internally.  If 
they  did  not  have  enough  work  that  required  teamwork,  then  they  could  not  have  worked  on  their 
teamwork  skills.  Teamwork  processes  may  be  measured  more  accurately  if  the  nodes  are  under 
more  pressure  and  need  to  use  their  teamwork  skills  in  order  to  accomplish  their  goals. 

Table  11 

Ratings  of  Behavior  Types  for  Each  Node 


Behavior 

Node 

Command  1 

Command  2 

Control  1 

Control  2 

Backup 

4.25 

4.90 

2.32 

1.80 

Communication 

5.19 

5.45 

2.96 

2.35 

Coordination 

4.67 

5.33 

2.64 

2.61 

Monitoring 

4.88 

5.30 

2.57 

2.19 

Team  Orientation 

4.88 

5.30 

2.69 

3.00 

Overall 

5.17 

5.10 

2.75 

3.29 

Note.  Ratings  based  on  7-point  scale,  1  =  very  poor  performance;  7  =  excellent  performance. 


In  the  interviews,  several  soldiers  made  the  recommendation  that  the  squadron  nodes 
needed  to  be  reconfigured.  The  most  common  observation  was  that  the  Control  1  node  function 
should  be  changed  to  control  all  of  the  reconnaissance  assets  within  the  squadron.  Several 
soldiers  thought  that  the  squadron  intelligence  officer  should  be  positioned  in  the  Control  2  node. 
The  Squadron  Commander  thought  that  a  staff  configured  into  geographically  separated  nodes 
would  decrease  the  overall  performance  of  the  staff  due  to  the  feeling  of  isolation  among  the 
staff  members. 

Sample  Result  -  Impacts  on  Perceived  and  Actual  Workload 

In  order  to  address  this  issue,  the  Individual  and  Team  Workload  survey  was  used.  First, 
participants  rated  their  own  individual  workload.  Next  they  rated  the  workload  of  each  node. 
Finally,  they  rated  the  workload  for  the  entire  staff.  When  rating  their  own  workload,  there  was 
no  consistent  pattern.  However,  there  was  a  general  downward  trend.  Mental  demand,  temporal 
demand,  effort,  and  frustration  decreased  as  the  BCR  DI  progressed;  performance,  however, 
remained  at  a  constant  level.  Figure  10  shows  this  general  trend  for  all  nodes.  This  result 
indicates  that  participants  felt  they  could  achieve  the  same  level  of  performance  with  less  effort 
the  more  familiar  they  became  with  the  reengineered  battle  command  concept  and  demands. 
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Note.  Ratings  based  on  1 1-point  scale,  1  =  very  low;  1 1  =  very  high. 

Figure  10.  Perceived  individual  workload. 

After  rating  their  own  workload,  participants  were  asked  to  rate  the  overall  workload  for 
each  node.  Again,  ratings  fluctuated  depending  on  the  activities  they  engaged  in  on  that 
particular  day.  In  general,  all  nodes  rated  everyone’s  workload  as  decreasing  over  time,  except 
Command  2,  which  rated  Control  1  and  2’s  workloads  as  decreasing  and  Command  1  and  2’s 
workloads  as  staying  consistent  over  time.  Table  12  shows  the  means  for  each  node  collapsed 
across  days. 

As  can  be  seen  in  the  table,  most  nodes  rated  their  own  workloads  somewhere  in  the 
middle.  Control  1  rated  themselves  as  higher  than  the  other  nodes  rated  it.  In  addition.  Control  1 
gave  the  highest  ratings  of  workload  for  all  the  nodes.  This  suggests  that  Control  1  may  not  have 
had  a  good  grasp  of  the  overall  workload,  and  of  what  the  other  nodes  were  doing.  The  other 
nodes  seem  to  have  a  fairly  good  idea  of  how  much  work  is  going  on  in  the  other  nodes.  This  is 
important,  because  if  the  nodes  are  aware  of  the  workload  of  others,  they  can  make  better 
judgments  on  when  someone  is  overloaded  and  may  need  help. 

Table  12 


Perceived  Workload  of  Nodes 


Rated  By 

Command  1 
Command  2 
Control  1 
Control  2 


Rating 


Command  1 
5:96 
5.59 
8.74 
7.40 


Command  2 
6:82 
7.22 
8.42 
7.50 


Control  1 
5?75 
6.22 
7.58 
4.79 


Control  2 
7A3 
6.30 
7.84 
6.51 


Note.  Ratings  in  bold  are  self-ratings.  Ratings  based  on  1 1 -point  scale,  1  =  very  low;  11=  very  high. 
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Finally,  participants  were  asked  to  evaluate  the  workload  of  the  entire  staff.  Figure  1 1 
shows  the  ratings  across  days  by  each  node.  As  can  be  seen  in  the  figure,  the  ratings  of  overall 
workload  generally  decrease  over  time.  Only  Control  1  rates  workload  as  increasing  toward  the 
end  of  the  BCR  HI.  Interestingly,  except  for  Control  1,  all  the  nodes  rate  the  level  of  overall 
workload  on  the  last  mission  as  equal  (6.00).  This  indicates  that  the  other  three  nodes  have  a 
common  understanding  of  their  workload. 


11 
10 
9 
8 
7 
6 
5 
4 
3 
2 
1 

TDX4  Pilot  1  Riot  2  Trial  1  Trial  2  Trial  3  Trial  4  Trial  5  Trial  6 
Mssion 

Note.  Ratings  based  on  1 1-point  scale,  1  =  very  low;  1 1  =  very  high. 

Figure  11.  Perceived  workload  of  entire  staff. 

■Summary  of  Performance  Evaluation  Findings 

The  sample  results  presented  above  provide  an  idea  of  the  way  the  different  measures 
were  used  to  study  staff  performance.  For  most  of  the  performance  issues,  more  than  one 
method  was  used  to  provide  corroboration  of  the  results  obtained.  Although  the  four  methods 
provided  the  expected  results,  some  improvements  can  be  made  for  future  research.  Below  is  a 
summation  of  each  evaluation  method. 

Aiitnmatftd  measurement.  The  data  collected  from  the  MMBL  DCA  were  cumbersome 
to  use.  Specific  data  points  had  been  requested  prior  to  the  start  of  the  BCR  HI,  but  were  not 
programmed  until  the  end.  Additional  data  collection  formats  had  to  be  designed  in  order  for  the 
data  to  be  pulled  from  the  system.  Once  the  data  were  obtained,  they  were  presented  along  with 
vast  numbers  of  other  data  points,  so  that  considerable  data  examination,  editing,  and  subsequent 
analyses  were  required.  Once  the  data  had  been  analyzed,  useful  results  were  obtained.  An 
example  of  an  automated  measure  analysis  is  presented  in  Appendix  F.  As  noted  previously, 
lack  of  immediate  BCR  HI  data  analyses  prevented  them  from  being  used  to  provide 
performance  feedback  to  the  training  unit. 

Surveys.  Surveys,  as  a  measurement  method,  were  easy  to  implement  although 
participants  complained  about  the  writing  chore.  The  particular  surveys  used  in  this  effort 
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should  not  necessarily  be  kept  for  future  research.  Instead,  surveys  that  provide  outcome  and 
process  data  that  can  be  more  clearly  linked  to  training  evaluation  and  battle  staff  performance 
should  be  developed  and  tested  in  future  research.  However,  surveys  will  not  be  appropriate  for 
obtaining  information  for  performance  feedback  unless  a)  a  method  for  quick  turnaround  of 
results  is  found;  and  b)  front-end  analysis  indicates  that  survey  data  can  address  questions  about 
the  unit’s  performance. 

Interviews.  Interviews  enabled  the  gathering  of  information  and  impressions  far  beyond 
what  was  obtained  in  surveys,  observations,  and  automated  means  during  BCR  HI.  Given  the 
current  difficulties  in  obtaining  and  using  automated  data,  a  great  deal  more  information  could 
have  been  collected  on  battle  staff  performance  by  means  of  interviews.  Like  surveys, 
interviews  were  not  useful  for  immediate  performance  feedback  purposes. 

Observation.  The  laptop-assisted  measurement  method  was  found  to  be  easy  to  use,  and 
the  data  capture  was  suitable  for  subsequent  analysis.  The  observers  using  the  ODCI 
recommended  adding  more  space  for  observer  notes  and  modifying  the  observation  categories. 
This  second  point  in  particular  is  important,  relating  to  the  previous  comments  on  front-end 
analysis.  Although  the  observation  categories  seemed  likely  candidates  as  sources  of 
information  about  team  performance,  they  were  less  useful  in  operation  than  had  been 
anticipated.  With  additional  enhancements  to  the  ODCI  system  that  would  allow  for  real-time 
transmission  of  observations  to  a  central  processing  site,  observations  could  be  a  valuable  input 
to  performance  feedback  sessions. 

Lessons  Learned  for  Future  Research 

As  a  research  and  development  effort,  this  project  has  as  a  major  product  the 
documentation  of  lessons  learned  These  lessons  are  addressed  to  several  audiences,  including 
developers  of  future  staff  training  and  researchers  conducting  advanced  C^I  systems  experiments. 
The  lessons  fall  into  four  categories^: 

•  Training  design 

•  Team  training  sessions 

•  Training  support  materials 

•  Evaluation  methods 

The  lessons  learned  are  divided  into  two  sections:  1)  those  that  are  specific  to  the  pilot 
implementation  of  the  prototype  staff  training  and  evaluation  package;  and  2)  those  that  are 
applicable  to  the  general  development  of  training  and  evaluation  programs. 


^  In  addition  to  the  observations  and  results  documented  in  the  previous  section,  lessons  learned  on  the  experiment 
design,  support  personnel  preparation,  documentation  of  TTPs,  battalion-level  staff  organizations,  and  design 
modifications  for  the  SC*  system  itself  were  provided  to  ARI  as  a  Memorandum  for  Record  (6  July  1999). 
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Pilot  Implementation  Lessons  Learned 


The  lessons  learned  from  the  pilot  implementation  of  the  prototype  training  and 
evaluation  package  are  summarized  in  Table  13.  A  detailed  discussion  of  each  lesson  follows 
the  table. 

Table  13 

Suimnary  of  Pilot  Implementation  Lessons  Learned 


Training  Design 

Revise  initial  orientation. 

Eliminate  movement  and  clustering  of  training  participants  during  function  training. 
Decrease  time  for  initial  orientation. 

Increase  fundamental  training  time. 

Increase  task  training. 

Decrease  time  for  TDXs. 

Integrate  extended  training  audience  into  all  training. 

Team  Training  Sessions 

Revise  information  management  training  session. 

Revise  title  of  training  sessions  to  reflect  current  Army  terminology. 

Training  Support  Materials 

Complete  existing  prototype  training  package. 

Develop  unit  preparation  plan. 

Develop  train-the-trainer  materials  for  team  training  sessions. 

Develop  train-the-trainer  materials  for  site  trainers. 

Evaluation  Methods 

Continue  the  design  and  development  of  automated  measures. 

Continue  development  of  ODCI. 

Continue  research  on  survey  measures  for  staffs. 


Training  Design 

The  prototype  staff  training  package  was  designed  as  a  hierarchical  progression  from  an 
initial  orientation  through  individual  and  small  group  training  to  collective  training  including 
team  training  sessions.  It  incorporated  aspects  recommended  in  the  training  literature,  including 
an  advance  organizer,  a  realistic  underlying  context,  and  both  formal  instruction  and  hands-on 
practice  opportunities  that  were  based  on  the  principles  of  training  vignettes  developed  in 
previous  ARI  sponsored  training  research  and  development.  All  of  these  aspects  should  be 
retained.  However,  additional  refinement  of  the  training  design  is  needed  in  three  areas: 
organization,  time  allocation,  and  training  audience. 
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Orpaniratinn.  The  training  was  structured  to  progress  through  five  levels.  The 
participants  concurred  with  the  structured  progression  and  thought  the  building  block  training 
sequence  was  effective.  They  also  agreed  with  the  concept  that  function-based  instruction 
assisted  them  in  the  development  of  SC'*  system  skills.  Detracting  from  this  overall  positive 
assessment  was  the  statement  from  many  participants  that  they  never  understood  the  goal  or  the 
performance  standard  for  each  training  level  or  how  the  training  levels  were  interrelated  until  the 
training  was  completed. 

To  address  this  shortcoming,  the  initial  orientation  needs  to  be  revised  to  clearly  establish 
that  there  is  progressivity  to  the  training  and  to  identify  individual  and  team  tasks,  with  their 
associated  standards  that  participants  are  required  to  master.  To  reinforce  the  standards,  the 
prototype  training  package  also  needs  to  incorporate  SC*  tool  skill  tests  that  are  embedded  in  the 
system.  Evaluated  functional  test  exercises  for  both  individuals  and  teams  within  the  staff  need 
to  be  developed,  as  well.  Participants  should  be  given  performance  feedback  and  any  required 
remedial  training  for  the  level  of  training  they  have  completed  before  they  progress  to  the  next 
level. 


Another  specific  concern  for  the  training  participants  was  the  consolidation  of 
participants  by  role  during  function  training  (Level  3)  which  was  idraitified  as  a  training 
detractor.  The  intent  was  to  have  those  participants  who  perform  the  same  function  in  different 
nodes  be  given  an  opportunity  to  develop  a  shared  understanding  of  how  to  exploit  the 
capabilities  of  the  SC"*  system  and  to  develop  SOPs  and  TTPs  among  themselves.  The  need  for 
this  type  of  training  was  predicated  on  a  model  in  which  an  SC*  equipped,  geographically 
dispersed  staff  would  distribute  its  products  or  processes  among  individuals  located  in  different 
nodes.  The  unit  staff  undergoing  the  training  in  the  pilot  implementation  of  the  prototype 
training  package  did  not  distribute  their  workload  in  this  fashion.  EiK^h  node  was  responsible  for 
a  product  and  rarely  did  individuals  in  different  nodes  collaboratively  work  on  a  product.  The 
shifting  of  participants  between  nodes  created  confusion  and  may  have  impeded  the  development 
of  intra-node  roles  and  functions.  Thus,  the  potential  value  of  having  participants  performing  the 
same  function  sharing  a  common  training  experience  was  negated. 

To  address  this  problem,  the  analysis  of  staff  processes  needs  to  be  refined  to  reflect  two 
modes  for  staff  operations:  1)  staff  product  and/or  process  responsibilities  will  be  assigned  to 
specific  nodes  or,  2)  the  workload  will  be  distributed  across  various  nodes.  In  either  case, 
physically  moving  training  audience  members  around  a  complex  training  site  should  be  avoided. 
If  the  refined  analysis  supports  continuing  collaborative  training  across  nodes  for  a  particular 
staff  process,  then  structured  practice  exercises  should  be  developed  to  allow  participants  to 
participate  in  collaborative  training  while  operating  from  their  assigned  node. 

Time  Allocation.  The  participants  indicated  that  the  5-day  p«iod  allocated  for  training 
was  sufficient,  but  the  time  for  classroom  orientation  and  instructiOT  should  be  limited  in  favor 
of  additional  practical  exercises  on  the  system.  Though  collective  task  training  utilized  the 
majority  of  the  time  allocated  for  training,  the  time  provided  was  still  insufficient  to  achieve  all 
training  objectives.  The  key  is  to  balance  the  time  required  to  teach  individuals  new  equipment 
skills  with  the  time  allocated  to  develop  techniques  and  procedures  that  integrate  these  skills  into 
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the  required  staff  functions  and  processes.  Training  on  new  systems  must  include  generous 
amounts  of  time  for  hands-on  practice  and  exploration. 

The  participants  stated  that  the  time  allocated  for  Level  1  training  (Initial  Orientation) 
was  too  long  for  the  information  provided.  Much  of  the  information  presented  was  at  too  high  a 
level  for  participants.  Additional  comments  revealed  that  the  time  devoted  to  multi-media  and 
slide  based  classroom  training  on  the  SC^  system  would  be  better  utilized  for  practice  on  the 
system.  The  schedule  provided  three  hours  for  the  orientation,  but  an  hour  would  be  sufficient. 
The  other  two  hours  should  be  added  to  fundamentals  (Level  2)  training  for  additional  practice 
exercises. 

The  time  allocated  for  functions  (Level  3)  training  seemed  sufficient,  but  the  participants 
indicated  that  the  four  hours  allocated  for  task  training  (Level  4)  seemed  insufficient.  Observers 
concurred  with  this  assessment.  They  noted  that  task  training,  which  uses  both  team  training 
sessions  to  train  higher  cognitive  skills  and  simulation  based  exercises  to  develop  inter-node 
roles,  battle  staff  TTPs,  and  SOPs,  requires  the  most  time.  The  first  team  training  session. 
Information  Management,  used  over  %  minutes.  Practicing  the  tasks  associated  with  the  Level  4 
structured  exercises,  such  as  establishing  a  whiteboard  conference,  diverted  more  time  from  the 
stated  training  objective  which  was  to  prepare  to  participate  in  the  TDXs.  More  time  is  required 
for  the  participants  to  fully  explore  their  roles  and  functions  and  achieve  the  task  training 
objective.  Level  4  should  be  expanded  to  12  hours.  This  increase  in  time  allows  full 
implementation  of  the  team  training  sessions. 

The  participants  observed  that  the  overall  pace  of  the  TDXs  (Level  5)  was  slow.  This 
was  due,  in  part,  to  the  failure  of  some  of  training  participants  to  achieve  all  of  the  objectives 
during  system  task  training.  This  limited  their  ability  to  develop  detailed  techniques  and 
procedures  associated  with  Level  4  roles  and  functions.  Observations  from  the  project  team 
indicate  that  reducing  Level  5  to  16  hours  by  transferring  eight  hours  to  Level  4  would  not 
adversely  affect  the  quality  of  the  Level  5  training. 

Training  Audience.  Although  the  prototype  staff  training  package  was  designed  for  the 
primary  battalion-level  staff  of  14  soldiers,  another  21  soldiers  were  required  to  support  some 
aspects  of  the  staff  training.  These  soldiers  also  needed  to  be  trained  on  the  operation  of  the  SC** 
system.  In  previous  ARI  structured  training  projects,  the  training  for  the  primary  participants 
and  other  personnel  required  to  support  the  primary  participants  was  conducted  along  separate 
time  lines.  However  this  project  required  the  supporting  participants,  at  the  company  and 
platoon  levels,  be  trained  on  the  same  timeline.  Additionally,  the  supporting  participants  were 
required  to  interface  with  the  primary  participants  for  specific  tasks  during  the  training  prototype. 
Many  of  the  21  soldiers  in  the  extended  training  audience  thought  that  they  were  inadequately 
trained  or  were  given  insufficient  opportunities  to  execute  challenging  operations  using  the  SC"* 
system  during  the  training  week. 

More  clearly  defined  roles  and  responsibilities  for  supporting  participants  need  to  be 
incorporated  into  the  prototype  training  package.  All  training  activities  need  to  include  both  the 
extended  training  audience  as  well  as  the  primary  audience.  Training  standards  for  the  extended 
audience  also  need  to  be  developed. 
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Team  Training  Sessions 


Staff  training  on  team  processes  such  as  decision-making,  shared  mental  models,  roles 
and  functions,  situational  awareness,  and  information  management  was  one  of  the  primary 
emphases  of  the  project.  The  pilot  implementation  of  the  prototype  training  package  provided 
the  project  team  with  an  opportunity  to  trial  five  specific  team  training  sessions:  Information 
Management,  Roles  and  Functions,  Pre-Action  Analysis,  Commander’s  Timeout,  and  Tactical 
Decision-Making  Debrief.  Comments  received  from  the  training  audience  members  indicate  that 
the  content  of  the  sessions  had  potential  for  being  useful,  and  that  sessions  conducted  as  part  of 
collective  training  exercises  would  be  valuable. 

The  Information  Management  Session,  which  was  identified  by  the  training  participants 
as  the  least  useful  training  session,  needs  further  research  and  development  both  as  to  its  content 
and  the  conditions  under  which  it  is  presented  to  a  training  audience.  Examples  used  diuing  the 
sessions  need  to  be  tailored  specifically  to  the  C"*!  system  used  by  the  staff.  Researchers  should 
also  determine  if  revising  the  titles  of  Ae  Pre-Action  Analysis,  Commander’s  Timeout,  Roles 
and  Functions,  and  Team  Decision-making  Debrief  training  sessions  to  reflect  contemporary 
Army  terminology  might  facilitate  wider  acceptance  of  these  innovative  training  sessions. 

Training  Support  Materials 

The  training  materials  used  for  the  pilot  implementation  of  the  prototype  training  package 
constituted  only  part  of  a  full-blown  training  support  package,  as  defined  in  U.S.  Army  Training 
and  Doctrine  Command  (TRADOC)  Regulation  350-70  (DA,  1999).  The  scenario  tactical 
materials,  simulation  files,  OPFOR  guides,  and  administrative  instructions  were  included.  Only 
partial  train-the-trainer  materials  for  the  initial  system  train-up  and  TDXs  were  included.  No 
materials  for  the  training  audience  were  prepared.  All  of  the  information  for  the  training 
audience  was  provided  either  through  RA-delivered  instruction  or  briefings  by  the  Training 
Director.  Because  there  were  no  training  audience  materials  and  only  partial  train-the-trainer 
materials,  the  information  presented  to  the  training  audience  was  perceived  (rightly)  as 
incomplete  and  inconsistent.  Additional  development  of  the  existing  prototype  training  package 
material  such  as  the  training  plan  outlines  and  practice  exercises  is  required  before  its  next  use. 
Based  on  the  result  of  the  pilot  implementation,  additional  material,  such  as  that  described 
below,  needs  to  be  developed  for  Ae  training  package  as  well. 

Unit  Preparation  Plan.  A  plan  needs  to  be  developed  to  assist  the  unit  to  take  advantage 
of  the  available  train-up  time.  As  in  the  BCR  HI,  a  short  briefing  for  the  full  training  audience 
should  be  provided  at  least  four  weeks  prior  to  the  training.  At  that  time,  the  training  audience 
members  should  be  told  the  objectives,  the  basic  schedule,  and  the  activities  that  will  take  place 
during  training  and  the  experiment.  They  should  be  told  what  they  should  have  accomplished 
before  they  arrive  for  the  train-up,  and  given  printed  or  CD-ROM  materials  illustrating  or 
guiding  the  preparation.  Topics  suitable  for  preparation  include  unit  SOPs,  techniques  and 
procedures  learned  from  previous  training  and  operations  (with  the  caution  that  they  will  likely 
develop  more  techniques  and  procedures  specifically  to  their  unit),  and  descriptions  of  team 
skills  that  they  can  begin  to  develop  or  think  about.  While  an  overview  of  the  C^I  system  and  its 
capabilities  would  be  useful,  the  unit  should  not  be  expected  to  get  familiar  with  the  system  prior 
to  arriving  at  the  site  unless  they  have  access  to  the  system  or  to  a  simulation  of  the  system. 


69 


Developers  of  this  preparation  plan  need  to  be  sensitive  to  other  competing  requirements  the  unit 
may  have  as  it  prepares  for  the  O'*!  system  training.  Unit  requirements  should  be  minimal, 
require  little  or  no  resources,  and  be  easily  mastered.  Appropriate  multi-media  material  that  may 
be  used  by  individuals  outside  of  their  normal  duty  place  and  time  should  also  be  explored  as  an 
option  for  assisting  the  unit  in  preparing  for  the  training. 

Team  Training  Session  Trainer  Materials.  These  train-the-trainer  materials  need  to  be 
developed.  A  model  for  the  contents  would  include  an  overview,  guidelines  for  the  facilitator, 
detailed  descriptions  of  the  sessions,  practice  exercises  for  each  type  of  session,  and  guidelines 
for  providing  feedback  to  the  participants.  A  key  component  of  the  train-the-trainer  material 
should  be  a  description  of  the  value  of  the  team  training  sessions  and  how  they  differ  from 
training  that  is  currently  being  done  for  staffs. 

Training  Site  Personnel  Materials.  These  train-the-trainer  materials  need  to  be  developed 
for  the  personnel  designated  to  implement  the  prototype  training  package  beyond  the  material 
that  was  developed  and  incorporated  into  the  training  package  for  the  Training  Director.  This 
material  should  include  an  overview  of  the  prototype  training  package,  a  detailed  description  of 
the  trainer’s  responsibilities,  structured  practice  exercises,  and  proficiency  test  materials.  In 
addition,  access  to  the  operator’s  manu^  for  the  O'*!  system  should  be  readily  available. 

Evaluation  Methods 


Considerable  research  remains  to  be  done  on  developing  both  performance  standards  and 
evaluation  methods  for  future  battle  staffs  operating  advanced  C"*!  systems.  While  surveys, 
observations,  and  interviews  were  used  to  gather  data  during  the  pilot  implementation,  the 
project  team  made  an  effort  to  develop  automated  measures  that  could  take  advantage  of  the 
analytical  power  and  processing  speed  of  advanced  C^I  systems  to  provide  real  or  near  real-time 
feedback  to  the  training  participants.  This  effort,  as  described  earlier,  was  only  partially 
successful.  Several  lessons  learned  during  this  project  may  be  of  value  to  other  researchers. 

Specifying  the  format  in  which  the  automated  measures  data  is  to  be  reported  will  aid  the 
programmer  extracting  data  from  the  C^I  system  and  simplify  read5dng  the  data  for  staff 
performance  feedback  sessions.  The  DCA  system  used  during  this  project  provides  the 
programmer  with  a  multitude  of  methods  to  manipulate  and  format  data.  Working  with  both  the 
operational  definition  of  an  automated  measure  and  the  specific  format  in  which  it  is  to  be 
reported,  the  programmer  will  be  better  able  to  meet  the  expectations  of  the  training  evaluator. 

Development  of  the  ODCI  based  on  laptop  computer  technology  or  personal  data 
assistant  technology  should  continue.  Strategies  to  deal  with  unexpected  events  or  with  planned 
events  that  did  not  occur  should  be  created.  The  ODCI  systems  should  be  linked  through 
networks  so  that  automated  processing  and  analysis  of  data  can  occur  simultaneously  with 
observation  recording  so  that,  like  the  automated  measurements,  feedback  can  be  provided  to  the 
staff  immediately  after  the  completion  of  their  training. 

More  research  is  required  to  locate  better  measures  of  staff  processes  that  are  appropriate 
for  use  during  surveys  than  were  used  during  this  project.  If  these  measures  do  not  exist,  then 
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they  should  be  developed  and  subjected  to  extensive  testing  to  determine  their  validity  and 
reliability. 

General  Lessons  Learned 

Table  14  summarizes  the  lessons  learned  during  this  project  that  may  apply  to  other 
training  design  and  development  efforts  related  to  advanced  C  I  systems  and  future  staffs.  A 
discussion  of  each  lesson  learned  follows. 

Table  14 

Summary  of  General  Lessons  Learned 


Trainin£  Design 

Determine  if  TDXs  and  Team  Training  Sessions  should  be  part  of  system  training. 
Explore  self-guided  individual  systems  training. 

Team  Training  Sessions 

Develop  strategy  to  increase  unit  leadership  acceptance  of  team  training  sessions. 
Incorporate  a  facilitator  from  outside  of  the  unit  to  implement  these  sessions. 

Develop  formal  team  training  feedback  sessions. 

Training  Support  Materials 

Develop  job  aids  for  participants. 

Evaluation  Methods 

Research  appropriate  relationship  between  outcome  measures  and  performance  measures. 
Expand  automated  measures. 

Develop  method  to  capture  emerging  TTPs  and  tactical  standing  operating  procedures 
(TACSOPs)  during  training. 


Training  Design 

In  designing  training  for  a  future  staff,  two  different,  but  interrelated,  techniques  (TDXs 
and  team  training  sessions)  were  employed  to  focus  on  improving  staff  metacognitive  skills, 
pMicularly  those  involved  with  supporting  the  commander’s  decision-making.  During  the  trial 
implementation  of  the  prototype  training  package,  the  challenges  and  stresses  associated  with 
learning  the  new  organization,  equipment,  tactics,  and  C^I  system  diminished  the  metacognitive 
aspect  of  the  TDXs  which  became  strictly  system  practice  training  exercises.  The  team  training 
sessions  were  completely  subsumed  by  other  considerations. 

An  analysis  is  required  to  determine  if  this  type  of  training  should  be  done  in  conjunction 
with  learning  to  function  as  a  reorganized  staff,  geographically  dispersed  into  small  tea^, 
operating  an  advanced  C^I  system  to  control  units  and  weapons  systems  that  may  exist  in  the 
future.  Researchers  should  look  to  implementing  these  sessions  in  a  less  demanding  systems 
training  environment  so  that  the  benefits  of  incorporating  them  into  staff  training  (which  are 
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described  in  the  theoretical  literature  and  which  were  recognized  by  some  of  the  training 
participants)  can  be  fully  realized. 

In  a  time  constrained  environment  where  training  participants  are  required  to  learn 
complex  procedures  to  operate  an  evolving  C^I  system  with  minimal  documentation,  detailed 
training  and  evaluation  activities  need  to  be  performance-based,  not  time-based.  After 
determining  the  minimum  individual  system  skills  needed  by  the  participants  to  be  able  to 
operate  effectively,  the  C^I  system  training  program  needs  to  be  as  flexible  and  open-ended  as 
possible.  It  should  be  self-guided  with  performance  standards  and  evaluation  activities 
incorporated  into  the  training  at  frequent  intervals.  By  placing  greater  emphasis  on  the 
participant’s  acquiring  basic  system  skills  early  in  the  training  program,  the  overall  unit  training 
program  can  be  enhanced.  Requiring  participants  to  engage  in  a  collective  activity  or  practice 
exercise  for  which  they  are  inadequately  trained  wastes  considerable  time  retraining  them  and 
holds  up  the  entire  unit.  Conversely,  participants  who  have  mastered  a  skill  should  be  able  to 
proceed  to  the  next  task  when  they  are  ready.  If  they  have  completed  the  training  program  in 
advance  of  the  rest  of  their  unit,  they  can  use  the  time  for  additional  practice  or  to  explore  other 
system  capabilities. 

Team  Training  Sessions 

Implementing  the  team  training  sessions  requires  the  cooperation  and  support  of  the  unit 
leadership,  particularly  the  battalion  commander.  This  individual  is  the  single  most  important 
influence  on  the  success  of  a  training  program.  An  innovative  training  strategy,  such  as  the  team 
training  sessions,  requires  that  the  leader  is  educated  in  advance  as  to  the  purpose  and  processes 
involved  with  the  training,  and  is  persuaded  to  wholeheartedly  support  it.  The  commander’s 
cooperation  and  support  are  dependent  upon  the  quality  and  perceived  utility  of  the  traimng,  and 
the  training  support  provided  by  the  training  director. 

Closely  associated  with  the  concept  of  unit  leadership  acceptance  of  the  training  program 
is  the  method  by  which  the  training  program  is  facilitated  within  the  unit.  The  prototype  training 
program  was  designed  to  be  initially  implemented  by  the  training  director,  and  transitioned  to  the 
battalion  commander  as  the  primary  staff  trainer.  The  design  also  required  the  commander  to  be 
a  key  participant  in  the  training.  This  design  should  be  reviewed  for  change  or  refinement.  A 
better  way  to  implement  the  team  training  sessions  might  be  to  have  a  trainer  from  outside  of  the 
unit,  such  as  a  higher  headquarters  commander,  an  experienced  peer  battalion  commander,  or  a 
trained  Senior  0/C  equivalent  in  rank  and  experience  to  the  battalion  commander,  to  facilitate 
these  sessions.  This  individual  needs  the  authority,  gained  by  subject  matter  expertise  and 
credibility,  and  supporting  training  materials  to  lead  such  sessions  after  completing  a  train-the- 
trainer  course  on  team  training  sessions.  Researchers  should  also  consider  implementing  team 
training  techniques  into  Army  professional  development  courses. 

Another  lesson  learned  is  that  team  training  sessions  need  to  have  formal  performance 
feedback  sessions  incorporated  into  them.  Even  if  the  sessions  provide  structured  practice 
exercises,  merely  providing  answers  to  the  exercises  is  insufficient.  In  a  complex  organization 
facing  challenging  problems,  there  is  usually  more  than  one  way  to  solve  a  problem.  By  using  a 
facilitator  to  guide  these  feedback  sessions,  the  training  participants  are  exposed  to  ways  to 
improve  their  performance  beyond  just  being  provided  the  right  answer  to  a  problem.  The 
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facilitator,  with  subject  matter  expertise  backed  by  experience,  can  suggest  other  ways  of  solving 
the  problem  and  provide  examples  to  guide  the  staff  in  improving  their  performance.  Such 
feedback  sessions  can  also  foster  among  the  participants  an  appreciation  that  their  team’s  overall 
efficiency  and  coordination  is  improving.  The  feedback  sessions  also  provide  a  method  by 
which  developers  can  assess  the  efficacy  of  particular  training  interventions  and  make 
improvements  without  having  to  directly  involve  the  training  participants  in  the  measurement. 

Training  Support  Materials 

Job  aids  should  be  a  primary  development  effort  for  any  training  program  involving  a 
complex,  advanced  O'*!  system.  These  should  be  developed  for  training  participants  to  portray 
the  locations  of  system  functions,  enemy  and  friendly  forces  capabilities,  training  objectives, 
node  responsibilities,  and  other  information  to  which  participants  need  to  refer.  As  doctrine  and 
TTPs  are  developed  for  future  staff  process  experimentation,  they  should  be  provided  as  ready 
references  to  training  participants  until  they  are  incorporated  into  standard  Army  doctrinal 
materials. 

Evaluation  Methods 

The  first  lesson  is  that  staff  performance  standards  developers,  and  by  extension, 
evaluation  method  developers  need  to  decide  whether  the  unit  outcome  of  the  staff  action  or  the 
processes  of  the  staff  should  be  the  focus.  If  the  staff  process  is  the  focus,  then  a  detailed 
analysis  of  the  processes,  which  includes  as  a  minimum  the  description  of  the  process  itself,  who 
is  involved,  what  tools  are  used,  and  what  products  are  generated  is  required  before  evaluation 
measures  and  methods  can  be  developed.  Even  if  unit  outcome  measures  will  not  be  the  primary 
tool  to  provide  staff  performance  feedback,  they  are  generally  easier  to  develop  than  process 
measures.  Consequently,  they  should  always  be  used  to  supplement  or  back  up  staff  process 
measures. 

Automated  measurement  of  staff  performance  continues  to  have  unrealized  potential. 
Additional  research  is  required  to  determine  which  advanced  digital  C^I  system  information  can 
be  used  to  measure  staff  processes  rather  than  just  outcomes.  This  information  should  then  be 
directly  importable,  without  additional  analysis  or  reformatting,  into  real  or  near  real-time  staff 
performance  feedback  sessions. 

A  method  to  document  emerging  TTPs  and  TACSOPs  needs  to  be  developed  prior  to 
implementing  prototype  training  for  future  staffs,  especially  in  those  instances  where  doctrine 
and  staff  processes  have  not  yet  been  established.  These  TTPs  and  TACSOPs  can  be  used  by  the 
Army  doctrine  community  to  refine  the  future  staffs  organization,  equipment,  and  operational 
doctrine. 


_  Summary  and  Conclusions 

The  prototype  staff  training  and  evaluation  package  developed  during  this  project  uses 
TDXs  and  team  training  sessions  based  on  cognitive  decision-making  theory  to  train  staff 
processes  important  in  the  information  age.  These  training  techniques  were  implemented  during 
the  Concept  Experimentation  Program  BCR  III  Experiment  conducted  by  the  MMBL.  The  BCR 
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in  provided  a  unique  opportunity  to  implement  this  prototype  training  package  since  the 
organization,  weapons,  equipment,  and  C^I  system  used  by  the  training  audience  were 
conceptual  models  of  what  might  be  used  by  the  Army  in  10-15  years. 

The  prototype’s  topics,  structure,  evaluation  plan,  and  methodology  for  development  are 
grounded  in  the  research  literature.  Development  of  the  prototype  training  and  evaluation 
package  followed  a  tried  and  proven  procedure  laid  out  by  TRADOC  and  previous  ARI  projects. 
Formative  evaluation  during  the  BCR  HI  provided  valuable  insights  upon  which  to  base 
revisions  to  the  design,  team  training  sessions,  training  support  materials,  and  evaluation 
methods.  The  revised  research  products  associated  with  the  prototype  training  package  for  the 
pilot  implementation  are  documented  in  the  five-volume  set  of  materials  entitled  Training  and 
Measurement  Support  Package,  Battle  Command  Reengineering  HI,  Mounted  Maneuver 
Battlespace  Lab  (Mounted  Maneuver  Battlespace  Lab,  1999c).  Researchers  interested  in  these 
volumes  can  access  them  through  the  Fort  Knox  MMBL.  Additional  trials  of  the  prototype 
training  and  evaluation  package  are  required  to  validate  its  efficacy  and  utility. 

Continued  development  of  the  team  training  sessions  is  essential  to  exploit  the 
technological  advances  in  information  processing  and  presentation  represented  by  the  SC^ 
system  employed  during  this  project.  There  are  two  parallel  phases  to  this  development  effort. 
The  first  phase  is  related  to  developing  C^I  system  expertise.  This  is  primarily  individual 
training,  done  either  in  professional  development  courses  or  through  on-the-job  training  within 
the  unit.  The  individual  staff  member,  by  virtue  of  his  or  her  assignment  selection,  is  tactically 
and  technically  qualified  in  terms  of  knowledge  and  skills  for  the  position  however,  he  or  she 
may  not  be  an  expert  with  the  particular  C'*!  system  used  by  the  staff.  The  unit  C*I  system 
training  needs  to  develop  or  maintain  expertise  within  the  staff  member. 

The  second  phase  in  developing  C‘*I  system  expertise  is  learning  how  to  function  as  a 
staff  consisting  of  teams  of  teams.  While  professional  development  schools  can  provide  a 
foundation  for  team  training,  it  is  the  unit  commander’s  responsibility  to  train  his  staff. 
Unfortunately,  there  are  few  resources  currently  available  to  assist  the  commander  in  assessing 
his  staffs  training,  especially  in  a  staff  equipped  with  emerging  advanced  C*I  systems  (Brown, 
1999).  Research  priority  should  be  given  to  developing  the  tools,  especially  embedded 
automated  measures  that  can  harness  the  data  processing  and  analytical  capabilities  of  the 
advanced  C^I  system  itself,  so  that  future  commanders  can  readily  assess  the  training  proficiency 
of  their  staffs. 

As  capabilities  of  advanced  C^I  systems  are  enhanced,  future  experimentation  should  lead 
to  further  development  of  training  methods  (particularly  for  developing  staff  teams)  that  target 
cognitive  skills  needed  on  the  digital  battlefield.  The  recently  adopted  TRADOC  Digital 
Learning  Strategy  envisions  a  5-step  process  to  train  commanders  and  staffs  (DA,  1998).  The 
prototype  staff  training  package  designed  and  developed  during  this  project  incorporates  several 
steps  of  that  strategy  and  can  serve  as  a  development  model  for  training  future  commanders  and 
battle  staff  to  gain  and  maintain  battlefield  dominance. 

The  literature  reviewed  during  this  project  indicates  that  tactical  decision-making  skills 
can  be  improved  with  training  that  focuses  on  pre-action  analysis,  examination  of  uncertainties 
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and  weaknesses  in  a  plan,  and  post-action  discussion  and  reinforcement  of  lessons  learned. 

Team  orientation  can  be  enhanced  by  training  focused  on  understanding  roles  and  functions, 
maintaining  situational  awareness,  and  attending  to  information  flow  management.  The 
prototype  staff  training  and  evaluation  package  discussed  in  this  report  includes  all  of  these 
elements. 

Lessons  learned  during  this  project  that  are  applicable  to  other  researchers  involved  in 
designing  training  for  future  staffs  include:  a  need  for  additional  analysis  to  see  if  higher 
cognitive  skill  training,  such  as  decision-making,  should  be  integrated  into  C*!  system  training  or 
conducted  separately;  developing  a  strategy  to  increase  unit  leadership  acceptance  of  team 
training  sessions  and  to  incorporate  an  outside  facilitator  to  implement  them;  and  the  provision  of 
individual  job  aids  during  training  for  complex  C'*!  system  procedures. 

Future  research  needs  to  be  directed  toward  designing  and  developing  evaluation 
methods  and  measures  for  staffs  operating  advanced  O'*!  systems.  Among  the  specific  areas  to 
be  researched  is  the  utility  of  system-embedded  evaluation  methods  and  the  appropriate 
relationship  between  outcome  measures  and  performance  measures.  Current  staff  evaluation 
methods  and  performance  measures  which  have  been  used  for  the  last  10  years  do  not  reflect  the 
greatly  increased  capabilities  that  emerging  C^*!  systems  will  provide  to  future  staffs.  Research 
should  begin  now  so  that  when  these  advanced  C^I  systems  are  operationally  ready,  performance 
evaluation  methods  and  measures  for  the  staffs  operating  them  will  be  ready  also. 

Limitations  in  the  methods  developed  and  formative  results  reported  are  readily 
acknowledged.  Prototype  methods  and  Concept  Experimentation  Programs  (CEPs)  are,  by  their 
very  nature,  exploratory  research  efforts.  In  particular,  the  futuristic  context  of  this  research 
resulted  in  certain  limitations.  The  prototype  training  and  assessment  methods  developed  for  the 
CEP  were  designed  for  staffs  operating  wiA  very  advanced  C4  systems  in  a  fully  digitized 
brigade  environment.  The  prototype  training  and  evaluation  methods  were  designed  to  exploit 
the  unique  capabilities  provided  by  a  distributed  interactive  simulation  test-bed.  Also,  as 
repeatedly  noted,  the  prototype  methods  were  not  fully  implemented  or  evaluated. 

Despite  these  limitations,  this  research  designed,  developed  and  implemented  prototype 
training  and  assessment  methods  for  information  age  staffs.  The  training  and  assessment  design 
was  based  on  a  review  of  the  current,  relevant  literature.  The  specific  methods  were  developed 
for  implementation  in  a  CEP  examining  the  tools  and  structure  for  future  battalion  staffs. 

Lessons  were  derived  concerning  both  specific  development  and  implementation,  and  conceptual 
design  of  these  methods.  Future  research  should  continue  the  effort  to  improve  training  and 
evaluation  methods  for  future  staffs. 
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Appendix  A 
List  of  Acronyms 

AA . assembly  area 

AAR . after  action  review 

ADA . Air  Defense  Artillery 

ANCOC . Advanced  Non-Commissioned  Officer’s  Course 

AO  AC . Armor  Officer’s  Advanced  Course 

AOB . Armor  Officer  Basic 

APC . Armored  Personnel  Carrier 

ARI . U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 

ARPA . Advanced  Research  Projects  Agency 

Arty . Artillery 

ARSI . Advanced  Research  Projects  Agency  Reconfigurable  Simulator  Initiative 

BCR . Battle  Conmiand  Reengineering 

BDA . Battle  Damage  Assessment 

BF . Battle  Force 

BLEFR . Battle  Lab  Experiment  Final  Report 

BLEP . Battle  Lab  Experiment  Plan 

BLUFOR . blue  forces 

BNCOC . Basic  Non-Conmiissioned  Officer’s  Course 

BPV . battlefield  planning  visualization 

C2 . command  and  control 

C2V . command  and  control  vehicle 

C^I . command,  control,  communications,  computer  and  intelligence 

CAS . Close  Air  Support 

CAU . crewman's  access  unit 

CCIR . commander’s  critical  information  requirements 

CEP . Concept  Experimentation  Program 

CIC . Combat  Information  Center 

CO  A . course  of  action 

CPU . Central  Processing  Unit 

CSS . combat  service  support 

CVCC . Combat  Vehicle  Command  and  Control 

DA . Department  of  the  Army 

DC . Deputy  Commander 

DCA . Data  Collection  and  Analysis  System 

DIS . distributed  interactive  simulation 

DTLOMS . doctrine,  training,  leader  development,  organizations,  materiel,  and  soldiers 

EEFI . . . Essential  Elenients  of  Friendly  Information 
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FBC . Future  Battlefield  Conditions 

PLOT . forward  line  of  troops 

FM . Field  Manual 

FOV . field  of  view 

FRAGO . fragmentary  order 

FSE . fire  support  element 

GCM . graphic  control  measures 

GOFER . Goals,  Options,  Facts,  Effects,  and  Review 

GUI . Graphical  User  Interface 

HMD . helmet-mounted  display 

HumRRO . Human  Resources  Research  Organization 

lAW . In  Accordance  With 

IB . international  border 

IFV . infantry  fighting  vehicle 

INTSUM . intelligence  summary 

LD . Line  of  Departure 

LOA . Limit  of  Advance 

MCOO . modified  combined  obstacle  overlay 

MDMP . military  decision-making  process 

METT-TC . mission,  enemy,  terrain,  friendly  troops,  time,  and  civilian  considerations 

MIBR . motorized  infantry  brigade 

MMBL . Mounted  Maneuver  Battlespace  Lab 

ModSAF . Modular  Semi-Automated  Forces 

MOP . Measure  of  Performance 

MUAV . Micro  Unmanned  Aerial  Vehicle 

NCO . Non-Commissioned  Officer 

NDM . Naturalistic  Decision-Making 

NLOS . non-line  of  sight 

0/C . observer/controller 

0/0 . on  order 

ODCI . Observer’s  Data  Collection  Instrument 

OIC . officer-in-charge 

OPFOR . opposing  force 

OPORD . operations  order 

PDA . Personalized  Decision  Analysis 

POI . program  of  instruction 

PVD . plan  view  display 
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RA . research  assistant 

RBC . Reengineered  Battle  Command 

RPD . Recognition  Primed  Decision 

R&S . Reconnaissance  and  Surveillance 

RSTA . Reconnaissance  Surveillance  Target  Acquisition 

SC* . surrogate  command,  control,  communications,  and  computer 

SITREP . Situation  Report 

SnTEMPS  ....Situational  Template 

SME . subject  matter  expert 

SOP . standing  operating  procedure 

SOV . Staff  Operations  Vehicle 

SPOTREP . Spot  Report 

STM . Staff  Training  in  Information  Management 

STO . Science  and  Technology  Objective 

TACSOP . tactical  standing  operating  procedure 

TACT . Team  Adaptation  and  Coordination  Training 

TARGETS . Target  Accepted  Responses  to  Generated  Events  or  Tasks 

TDG . Tactical  Decision  Game 

TDX . tactical  decision-making  exercise 

TF . Task  Force 

TRADOC . U.S.  Army  Training  and  Doctrine  Command 

TSP . training  support  package 

TTP . tactics,  techniques,  and  procedures 

TTS . Team  Training  Sessions 

UAV . unmanned  aerial  vehicle 

UGV . unmanned  ground  vehicle 

USAARMC  ...U.S.  Army  Armor  Center 
USMC . U.S.  Marine  Corps 

VTC . video  teleconference 

WARNO . warning  order 
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Appendix  B 


Sample  Materials  from  Volume  1  of  the 
Training  and  Measurement  Support  Package 

This  appendix  provides  a  sample  set  of  the  materials  contained  in  Volume  1:  Overview 
and  Front-End  Analysis  for  Training  and  Measurement.  Volume  1  is  part  of  the  five  volume  set. 
Training  and  Measurement  Support  Package,  Mounted  Maneuver  Battlespace  Lab,  Battle 
Command  Reengineering  III  (Training  and  Measurement  Support  Package,  1999),  which 
documents  the  work  performed  in  the  design  of  a  prototype  training  program  and  evaluation 
package. 

The  purpose  of  this  appendix  is  twofold.  First,  it  provides  the  reader  with  samples  of  the 
front-end  analysis  products  used  in  the  development  of  the  prototype  training  package. 

Secondly,  the  appendix  provides  a  general  overview  of  the  topics  and  materials  found  in  Volume 
1 .  Training  developers  and  others  who  are  interested  in  the  training  of  information  age  brigade 
and  battalion  level  battle  staffs  may  find  this  information  of  value  to  them. 

The  table  below  is  the  complete  table  of  contents  for  Volume  1.  The  first  column  in  the 
table  provides  the  title  for  each  product.  The  second  column  (where  indicated)  states  whether 
this  section  contains  a  partial  or  a  complete  sample  of  the  product  and  the  last  column  identifies 
the  page  number. 
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Node  Responsibility  and  Individual  Job  Responsibility  Descriptions 


Command  1  (Command  Group):  Provides  the  Commander  with  the  capability  to  accurately  assess  the 
combat  situation,  continually  track  critical  events,  make  timely  decisions,  and  transmit  his  decisions  to  his 
subordinate  commanders  and  higher  headquarters. 

■Squadron  Commander:  Commands  the  Sqdn,  initiates  planning  and  decision-making,  approves 
WARNOs/FRAGOs/OPORD,  leads  wargaming  and  rehearsals. 

Effects  Officer:  Develops  Sqdn  fire  plan,  monitors  and  manages  battlefield  effects,  initiates  artillery 
and  mortar  calls  for  fire,  tracks  enemy  Battle  Damage  Assessment  (BDA),  monitors  Sqdn  operational 
status. 

Enemy  Operations  Officer:  Analyzes  Bde  R&S  plan,  develops  Sqdn  R&S  plan,  directs  scouts  and 
sensors,  tracks  enemy  BDA. 

Command  2  (Deputy  Commander):  Coordinates  and  synchronizes  combat  support  and  combat  service 
support  activities.  Assumes  responsibilities  of  Command  1  if  BCVl  is  damaged  or  destroyed. 

Deputy  Commander:  Chief  of  Staff.  Monitors  battle,  manages  terrain  deconfliction,  monitors  Sqdn 
status. 

Operations  Officer:  Coordinates  with  Bde  for  recon  LD  and  LOA,  assists  with  Sqdn  R&S  plan, 
manages  satellite  imagery,  manages  Close  Air  Support  (CAS),  monitors  enemy  movement  and  BDA. 

Operations  NCO:  Creates  operational  graphics  files,  monitors  fires  effects, 
friendly  operations,  and  Sqdn  status. 

Control  1  or  Control  2  (in  Current  Ops  Mode):  Synchronizes  current  combat  operations.  Develops 
branches  to  current  operations.  Assumes  responsibilities  of  Command  2  or  Control  2  if  BCV2  or  SOV2 
is  damaged  or  destroyed. 

Battle  Captain:  Manage  OPORD  development;  receive,  develop,  coordinate  &  distribute  WARNO  to 
Sqdn;  initiates  whiteboard  conference;  distribute  approved  OPORD  to  Sqdn. 

Friendly  Operations  NCO:  Monitor  current  TF  Class  nW  and  slant  information,  track  CCIR. 

Enemy  Operations  NCO:  Track  CCIR  and  Essential  Elements  of  Friendly  Information  (EEFI),  track 
frontline  trace,  supervise  sensor  NCO  during  sensor  missions. 

Sensor  NCO:  Operate  sensors  lAW  Sqdn  R&S  plan,  ensure  Sqdn  has  latest  enemy  info. 

Control  1  or  Control  2  (in  Future  Ops  Mode):  Plans  future  operations  while  monitoring  current 
situation  for  impact  on  future  operations  and  maintaining  current  enemy  situation.  Assumes  functions  of 
Control  1  if  SOVl  is  damaged  or  destroyed. 

Battle  Captain:  Review  R&S  plan  overlays  and  fire  plan  overlays,  review  OPORD  graphic  control 
measure  (GCM). 

Friendly  Operations  Officer:  Maintain  friendly  current  situation,  track  CCIR,  monitor  fires  effects, 
monitor  Sqdn  maintenance  and  supply  status,  track  Sqdn  repair  and  resupply,  distribute  Sqdn  status, 
manage  future  CSS  requirements. 

Enemy  Operations  NCO:  Track  CCIR  and  EEFI,  track  frontline  trace,  supervise  sensor  NCO  during 
sensor  missions. 

Sensor  NCO:  Operate  sensors  lAW  Sqdn  R&S  plan,  ensure  Sqdn  has  latest  enemy  info. 
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Performance  Analysis 
(Mission  Analysis  and  Wargaming) 


1.2  Quick  Assessment 

1 .3  Bde  order  sent  to  other  E-mail  &  White  1.3  Receives  Bde  order  with  E-mail  &  White  1.3  Receives  Bde  order  with  E-mail  &  White  1.3  Receives  Bde  order  E-mail  &  White 

nodes  Board  graphics  Board  graphics  Board  with  graphics  Board 


Performance  Analysis,  cont. 
(Mission  Analysis  and  Wargaming) 


1 .1 2  Car  approves  WARNO' 
1  and  sends  to  Current  Ops 


SC4  Tool  Descriptions 


SC4  TOOLS  DESCRIPTION 


File 

Provides  options  for  file  and  preferences 

New  Scenario 

Clears  PVD  of  all  overlays  and  allows  for  construction  of  new  scenario. 

Load/Save/Delete  User  Preferences 

Set  up  in  User  Preferences  toolbar  and  save. 

Save  Overlay  to  File 

Save  an  overlay  to  a  file. 

Load  Overlay  File 

Load  an  overlay  for  use  with  the  Overlay  Ecf/for  toolbar  button 

Delete  Overlay  File 

Permanently  deletes  overlays  from  the  (CPU).  Do  not  use  unless  directed. 

Quit 

Terminates  MODSAF  program.  Do  not  use  unless  directed. 

Map  Scale  Used  to  change  map  scale  from  1 :500,000  down  to  1 :250.  Force  the  map  display  to  zoom  in  or 

out  to  fit  a  desired  scale,  _ 


Map  Features  _ Turn  on  and  off  various  terrain  features  to  provide  the  amount  and  type  of  info  required. 


Show  As 

Change  how  friendly  and  enemy  units  are  displayed  on  the  PVD. 

Vehicle  Pictures 

Displays  individual  vehicle  pictures. 

Vehicle  Icons 

Displays  vehicles  as  individual  icons. 

Agg  Platoons 

Displays  platoon  aggregate  units. 

Agg  Companies 

Displays  company  aggregate  units. 

Special 

Toggle  Status  Display 

Used  to  select  icons  (blue  or  red)  and  receive  either  a  SPOTREP  (enemy)  or  SITREP  (friendly). 

Freeze  Display 

Allovre  the  current  situational  awareness  display  to  be  frozen  and  saved  to  use  as  a  whiteboard 
tool. 

Show  Editor 

Forces  space  at  the  bottom  <rf  the  PVD  to  remain  open  so  that  when  a  toolbutton  is  enabled  the 
map  is  not  redrawn. 

TOOLBAR  BUnONS 

Buttons  down  left  side  of  screen. 

Text  Editor 

Place  text  in  a  selected  overlay. 

Line  Editor 

Place  lines  in  a  selected  overlay. 

Area  Editor 

Place  areas  in  a  selected  overlay. 

Point  Editor 

Place  points  and  unit  symbols  in  a  selected  overlay. 

PVD  Controls 

Limited  configuration  display  of  the  PVD. 

User  Preferences 

Establish  how  information  is  displayed  on  the  PVD.  Do  not  use  unless  directed. 

Stealth  Control 

Place  the  stealth  viewer  at  a  specific  location  on  the  map. 

Delete  Tool 

Delete  objects  from  overlays. 

Overlay  Editor 

Create  new  overlays,  and  turn  on  and  off  overlays  on  the  PVD. 

Call  For  Fire  Tool 

Send  a  call  for  fire  directly  to  the  FDC  without  using  voice  communication. 

Terrain  Tool 

Analyze  terrain  for  intervisibility  and  to  generate  a  terrain  profile. 

Field  of  View  Tool 

Analyze  terrain  for  FOV  for  both  sensors  and  weapon  systems  and  generate  an  overlay  of  that 
FOVforusein  planning. 

Snail  Display 

Analyze  unit  movement  over  selected  periods  of  time  and  generate  an  overlay  of  that  unit 
movement  for  use  in  planning. 
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SC4  Functions 


Four  basic  SC4  functions  were  identified,  and  a  fifth  category  was 
defined  to  include  functions  that  are  not  included  in  the  SC4  system, 
but  that  add  to  the  commander’s  situational  awareness. 

Each  function  is  defined  below.  The  functions  are: 

•  Common  Relevant  Picture  Management, 

•  Information  Management, 

•  Conference  Management, 

•  Orders  Production,  and 

•  Other  Support  Functions  (e.g.,  UAV,  Battle  Planning  and 
Visualization  Display  [BPV]). 

These  functions  served  as  the  foundation  for  Level  3  training. 


Common 

Relevant 

Picture 

Management 


The  tools  and  procedures  that  allow  the  commander  and  his  staff  to 
develop  a  clear  understanding  of  his  current  situation  in  relation  to  the  enemy 
and  environment.  Two  areas  within  this  function  are  SITTEMPs  and  CCERs. 
Activities  includes: 

•  getting  different  kinds  of  information  from  the  PVD  display, 

•  viewing  one  or  more  overlays  on  the  PVD, 

•  using  tools  for  terrain  analysis  and  fields  of  view, 

•  viewing  history  of  troop  movements  (past  60  minutes), 

•  projecting  enemy  movements, 

•  setting  up  event  boxes  for  automatically  monitoring  critical  events, 
and 

•  using  the  PVD  and  other  tools  to  answer  commander’s  requirements 
for  information. 


The  activities  emphasized  are  slightly  different  for  each  of  the  following 
participant  groupings: 

•  Effects  Officer  -  Battle  damage,  positioning. 

•  Operations/Friendly  Ops  -  Status  of  friendly  forces,  staff  estimates. 

•  Enemy  Ops  -  Monitoring,  interpreting,  predicting  enemy  activity. 


Continued  on  next  page 


B-6 


SC4  Functions,  Continued 


Information 

Management 


Conference 

Management 


The  tools  and  procedures  that  allow  the  commander  and  staff  to  gain  a 
fuller  understanding  of  the  current  situation  by  accessing,  processing, 
disseminating,  and  storing  information  that  cannot  be  readily  projected 
onto  a  common  relevant  picture  display  (i.e.,  the  PVD).  This  function 
covers  status  reports,  spot  reports,  and  electronic  messaging.  Activities 
include: 

•  reviewing  PVD  tools  and  screen  arrangement, 

•  getting  reports  of  friendly  and  enemy  situation,  status,  and  activity,  and 

•  managing  e-mail  capabilities  aiiid  features. 

The  activities  emphasized  are  slightly  different  for  each  of  the  following 
participant  groupings: 

•  Vehicle  Commanders  -  Access  to  all  information  (including  stealth 
view)  while  not  overloading  with  all  details  all  the  time. 

•  Friendly  Operations  Staff  -  Friendly  forces  activities,  situation  reports, 
and  status. 

•  Enemy  Operations  Staff  -  Enemy  activities  and  projections. 

•  Sensor  Operators  -  Comparing  information  from  multiple  sources  to 
aid  in  interpretation. 


The  tools  and  procedures  that  allow  the  commander  to  communicate  his 
decisions,  guidance,  and  intent  in  situations  where  the  commander,  his  staff, 
and  subordinate  commanders  are  geographically  dispersed.  These  tools  also 
facilitate  the  commander’s  ability  to  supervise  and  adjust  his  subordinate 
forces’  execution  to  ensure  compliance  with  his  intent.  Activities  includes: 

•  using  the  whiteboard  for  inter-node  planning,  wargaming,  and 
rehearsals  and 

•  using  video  teleconferencing  (VTC),  with  and  without  the  whiteboard. 


Continued  on  next  page 
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SC4  Functions,  Continued 


Orders 

Production 


Other  Support 
Functions 


The  tools  and  procedures  that  allow  the  commander  and  his  staff  to 
communicate  the  commander’s  decisions,  guidance,  and  intent  to  superior  and 
subordinate  commanders  and  staff  in  a  commonly  understood  format  in 
situations  when  there  is  sufficient  time  for  the  commander  and  staff  to 
formally  record  the  decision-making  process.  It  also  includes  transmitting  the 
results  to  subordinate  conunanders  in  time  for  them  to  understand  the 
commander’s  intent  without  impacting  on  the  unit’s  preparatory  activities. 

Overlay  production  is  done  on  the  PVD.  The  rest  of  the  order  will  generally 
be  done  via  the  whiteboard. 


The  tools  and  procedures  that  are  not  included  in  the  SC4  system,  but 
that  add  to  the  commander’s  situational  awareness.  Examples  of  these 
tools  include: 

•  UAV  Sensor  Control, 


•  BPV, 

•  Satellite,  and 

•  Netscape  Navigator. 
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SC4  Tools  by  Functions  Crosswalk 


Common 
Relevant 
Picture  Mgt 


Conference  Information  Orders 
Mgt  Mgt  Production 


Other 


PVD 


FUe 


New  Scenario 


Load/Save/Delete  User  Preferences 


Save  Overlay  to  File _ 


Load  Overlay  File  _ 


Delete  Overlay  File 


Quit  _ 


Map  Scale 


Map  Features 


Show  As 


Vehicle  Kctures 


Vehicle  Icons 


Agg  Platoons _ 


Agg  Companies _ 


Special 


Toggle  Status  Display _ 


Freeze  Display _ 


Show  Editor 


TOOLBAR  BUTTONS 


Text  Editor 


Line  Editor  _ 


Area  Editor 


Point  Editor 


PVD  Controls 


*  User  Preferences 


Stealth  Control 


Delete  Tool  _ 


Overlay  Editor 


FSE  (Call  For  Fire)  Tool 


Terrain  Tool 


Field  of  View  Tool 


Snail  Display  _ 


Forward  Line  of  Troops  (FLOT) 
Disnla 


PIR/CCIRTool 


Mouse  Button  Control 


Information 


Pan 


Environmental  Information 


Functions  by  Responsibilities  Crosswalk 


Tool  by  Training  Level  Summary 


New  Scenario 


Load/Save/Delete  User  Preferences 


Save  Overlay  to  R!e 


Load  Overlay  Rle 


Delete  Overlay  File 


Quit 


Map  Scale 


Map  Features 


Show  As 


Vehicle  Pictures 


Vehicle  icons 


Agg  Platoons 


Agg  Companies 


Special 


Toggle  Status  Display 


Freeze  Display 


Show  Editor 


TOOLBAR  BUTTONS 


Text  Editor 


Line  Edrtor 


Area  Editor 


Point  Editor 


PVD  Controls 


•  User  Preferences 


Stealth  Control 


Delete  Tool 


Overlay  Editor 


FSE  (Call  For  Fire)  Tool 


Terrain  Tool 


Field  of  View  Tool 


Snail  Display 


Forward  Line  of  Troops  (FLOT)  Display 


PIR/CCIRTod 


Fife  Toot 


Snap  Tool 
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Appendix  C 


Sample  Materials  from  Volume  2  of  the 
Training  and  Measurement  Support  Package 

This  appendix  provides  a  sample  set  of  the  materials  contained  in  Volume  2:  Initial 
Orientation  and  Train-Up.  Volume  2  is  part  of  the  five  volume  set.  Training  and  Measurement 
Support  Package,  Mounted  Maneuver  Battlespace  Lab,  Battle  Command  Reengineering  III 
(Training  and  Measurement  Support  Package,  1999),  which  documents  the  work  performed  in 
the  design  of  a  prototype  training  program  and  evaluation  package. 

The  purpose  of  this  appendix  is  twofold.  First,  it  provides  the  reader  with  brief  samples 
of  the  orientation  and  train-up  materials  that  were  mentioned  in  the  report  for  use  in  conducting 
the  prototype  training.  Secondly,  the  appendix  provides  a  general  overview  of  the  topics  and 
materials  found  in  the  Volume  2.  Training  developers  and  others  who  are  interested  in  the 
training  of  information  age  brigade  and  battalion  level  battle  staffs  may  find  this  information  of 
value  to  them. 

The  table  below  is  a  general  table  of  contents  for  Volume  2.  The  first  column  in  the  table 
provides  the  title  for  each  product.  The  second  column  (where  indicated)  states  whether  this 
appendix  contains  a  partial  or  a  complete  sample  of  the  product  and  the  last  column  identifies  the 
page  number. 


Volume  Title 

Appendix 

Sample 

Appendix 

Page 

Overview 

Training  Schedule 

Not  Provided 

- 

Level  1:  Initial  Orientation 

Training  Plan  Outline 

Not  Provided 

— 

Initial  Orientation  Slides 

Partial 

C-2 

Level  2;  Fundamentals  Training 

Training  Plan  Outline 

Partial 

C-4 

Signal  Operating  Instructions  Extract 

Not  Provided 

— 

Structured  Practice  Exercise 

Partial 

C-5 

Level  3:  Functions  Training 

Training  Plan  Outline 

Not  Provided 

Signal  Operating  Instructions  Extract 

Not  Provided 

— 

Structured  Practice  Exercise 

Not  Provided 

— 

Level  4:  Tasks  Training  and  Practice 

Training  Plan  Outline 

Not  Provided 

— 

Attachments  to  Level  4 

Not  Provided 

-- 
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ADVANCE  ORGANIZER  FROM  INITIAL  ORIENTATION 


Advance  Organizer  from  Initial  Orientation,  cont. 
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i^i 

“  Initial  Orientation  -  Level  1 

An  Advance  Organizer  Approach 

Current  BCR  III 


--  Staff 
Structure 

Commander,  primary  and 
special  staff 

Commander  wHh  Command 
group  and  three  C2  nodes 

-  METT-TC 

Missions,  Enemy  Doctrine, 
Sensors,  etc. 

W 

Missfohs,  Enemy  pocMne  - 
and  Capabllfty,  Sensbm,  etc^ 

-  CPs  and 
Equipment 

Main,  Rear,  TAC; 
radios,  ASAS,  MCS... 

]i 

C2  func^hs;  SC4  7 

-  Planning 
Processes 

Military  Decision-Making 
Process,  plans  cell 

W 

currentand  fut^  ops  nodes 

Preview  of  Experiment  Process,  Training  (TDX,  Tasks, 
Functions,  Fundamentals)  for  2  hours  on  Day  1 


:NTEGRAT0R  of  MOUHTED  B&TTLE3PACK 


Training  Plan  Outline 


Level  2TtBining  Plan  Outline^  Continued 


Level! 

instruction, 

computer 

display 

management 


3.  Computer  display  management  (30  min.,  SPE  10  minutes) 

a.  Bri^y  review  computer  monitors  and  uses 

b.  Identify  the  Workspace  Menu 

□  Show  how  to  access 

□  Briefly  describe  run  ModSAF  arid  C2  (bring  up  the  three  basic 
GUI  windows) 

c.  Name  and  describe  the  three  basic  display  windows 

□  Describe  graphic  user  interface  (GUI)  windows 

Plan  View  Display  (PVD) 

Netscq)e  Mail  (E-mail) 

Simplicity  ™  Whiteboard  (Whiteboard) 

□  Note  to  avoid  closing  any  GUI  window 

d.  GUI  window  manipulation 

□  Show  how  to  bring  forward,  maximize,  restore,  move,  resize, 
minimize,  restore  minimized  window 

0  Show  how  to  set  up  the  suggested  optimal  window  arrangement 


e.  Point  out  general  use  for  the  three  GUI  windows 

D  PVD  provides  the  common  situational  awareness  picture  (e.g., 
terrain,  overlays,  call  for  fire) 

0  email  provides  written  communication  between  nodes 
□  whiteboard  provides  capability  to  conference  on-line  between 
nodes  with  text  and  graphic  files 
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Structured  Practice  Exercise 


Set  B:  Computer  Display  Management  (1  of  3) 

1 .  The  Workspace  Menu  contains  options  to  re-start  GUI  windows.  To  obtain  the  Workspace  Menu, 
right  click  anywhere  on  the  monitor  screen  where  there  is  not  an  open  window 

2.  Bring  up  the  GUI  windows;  PVD,  Whiteboard,  E-mail  (may  need  to  close  them  first) 

3.  To  remove  the  workspace  window  from  the  monitor  screen,  left  click  once  anywhere  on  the 
monitor  screen 

4.  To  bring  the  Netscape  Mail  window  forward,  which  is  used  to  send  or  receive  e-mail,  click  on  the 
header  bar  at  the  top  of  the  window 

5.  To  see  the  terrain  map,  click  on  the  header  bar  of  the  PVD  (ModSAF)  window 

6.  Maximize  the  PVD  (ModSAF)  window  size  by  clicking  on  the  maximize  tool  button  located  on 
the  far  right  at  the  top  of  the  window 

7.  Restore  the  PVD  window  screen  size  by  clicking  on  the  same  button 

8.  Click  and  hold  on  the  top  header  bar  of  the  PVD  window  and  move  the  window  to  the  bottom  of 
the  monitor  screen 

9.  Bring  forward  the  Whiteboard  window  by  clicking  on  the  header  bar  of  the  window 

10.  Resize  the  whiteboard  window  by  grabbing  a  comer  of  the  window  with  the  mouse  and 
expanding  the  window 

1 1 .  Minimize  the  whiteboard  window  by  clicking  on  the  minimize  button  (next  to  the  maximize 
button)  at  the  top  of  the  window,  the  ICON  labeled  Whiteboard  will  appear  on  the  left  side  on  the 
monitor  screen 

12.  Should  the  whiteboard  ICON  be  covered  by  another  window,  move  that  window  out  of  the  way 
by  clicking  and  holding  on  the  top  header  bar 

13.  Click  once  on  the  Whiteboard  ICON  to  obtain  menu  options 

14.  Click  on  the  Restore  option  to  bring  up  the  whiteboard  window 

15.  Minimize  the  whiteboard  window,  click  the  minimize  button  at  the  top  of  the  window 

16.  Double  click  on  the  Whiteboard  ICON  to  restore  the  window 

17.  Assist  the  training  audience  in  returning  the  monitor  screen  back  to  the  original  recommended 
window  arrangement.  (All  three  header  bars  can  be  seen  at  the  same  time,  with  the  Netscape 
window  to  the  top  left,  whiteboard  to  the  top  right,  and  PVD  just  below  those  in  the  center  of  the 
screen) 
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Sample  Materials  from  Volume  3  of  the 
Training  and  Measurement  Support  Package 

This  appendix  provides  a  sample  set  of  the  materials  contained  Volume  3:  Tactical 
Decision-Making  Exercises  and  Team  Training  Sessions.  Volume  3  is  part  of  the  five  volume 
set,  Training  and  Measurement  Support  Package,  Mounted  Maneuver  Battlespace  Lab,  Battle 
Command  Reengineering  HI  (Training  and  Measurement  Support  Package,  1999),  which 
documents  the  work  performed  in  the  design  of  a  prototype  training  program  and  evaluation 
package. 

The  purpose  of  this  appendix  is  twofold.  First,  it  provides  the  reader  with  brief  samples 
of  selected  training  materials  Aat  were  mentioned  in  the  report  designed  and  developed  to 
conduct  the  TDXs.  Secondly,  the  appendix  provides  a  general  overview  of  the  topics  and 
materials  found  in  the  volume.  Training  developers  and  others  who  are  interested  in  the  training 
of  information  age  brigade  and  battalion  level  battle  staffs  may  find  this  information  of  value  to 
them. 


The  table  below  is  a  general  table  of  contents  for  Volume  3.  The  TDX  sample  materials 
are  taken  from  TDX  1 ,  Mission  Analysis  and  Wargaming.  Although  the  TDX  Workstation 
Guidelines  include  instructions  for  various  OPFOR  or  BLUFOR  workstations,  only  one 
guideline  (OPFOR  workstation)  is  provided  as  an  example. 

The  first  column  in  the  table  provides  the  title  for  each  product.  The  second  column 
(where  indicated)  states  whether  this  appendix  contains  a  partid  or  a  complete  sample  of  the 
product  and  the  last  colunm  identifies  the  page  number. 


VoluineTitle 

Appendix 
r -Sample 

Appendix 

Page 

Overview 

Not  Provided 

— 

Training  Director  Guidelines 

Partial 

D-3 

TDX  Structure* 

Complete 

D-4 

TDX  1:  Mission  Analysis  and  Wargaming 

TDX  Overview 

Complete 

D-5 

TDX  Event  Description 

Partial 

D-6 

TDX  Event  Guide 

Partial 

D-7 

TDX  Workstation  Guidelines 

Partial 

D-8 

Tactical  Materials 

Complete 

D-9 

TDX  Plan  Sheets 

Partial 

D-10 

TDX  2:  Rehearsal 

(similar  materials  as  TDX  1) 

Not  Provided 

TDX  3:  Execution  of  a  Squadron  Branch 

(similar  materials  as  TDX  1) 

Not  Rovided 

(table  continues) 
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Table  5  (Continued) 


Volume  Tide 

Appendix 

Sample 

Appendix 

Page 

TDX  4:  Execution  of  a  Squadron  Sequel 

(similar  materials  as  TDX  1) 

Not  Provided 

-- 

Attachments 

Signal  Operating  Instructions  Extract 

Not  Provided 

Team  Training  Sessions  Slides 

Partial 

D-11 

Tactical  Materials  for  Level  5 

Not  Provided 

— 

Acronyms 

Not  Provided 

— 

Note.  The  item  marked  with  an  asterisk  (♦)  is  not  a  separate  document  in  Volume  3.  It  is  part  of  the  Volume  3 
Training  Director  Guidelines  and  is  provided  to  summarize  the  general  TDX  phases  and  associated  activities. 
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TDX  Structure 


There  are  three  phases  to  a  TDX:  Pre-execution,  Execution,  and  Post- 
TDX  structure  execution.  Descriptions  of  the  phases  and  the  sequence  of  activities  in  each 
TDX  are  provided  in  the  following  table. 


Phase 

Activity 

Description 

Materials 

Pre-Execution 

TDX  Description 

Training  Director  provides  a  brief 
description  of  the  TDX  and  the 
exercise  training  objective(s).  If 
necessary,  the  starting  locations  for 
squadron  units  is  also  provided. 

Training  Director 
Guidelines 

TDX  Overview 

Pre-Action  Analysis 
(TTS  Session) 

Training  Director  facilitates  Pre- 
Action  Analysis  initially.  Goal  is  to 
have  Squadron  Commander 
facilitate  this  session. 

TTS  SUdes 

Execution 

Begin  TDX 

Training  Director  uses  the  TDX 
event  guide  to  control  and  cue  unit 
training. 

TDX  Event  Guide 

TDX  Guidelines 

Commander’s  Timeout 

(ITS  Session) 

Training  Director  facilitates 
Commander’s  Timeout  initially. 

Goal  is  to  have  Squadron 
Commander  facilitate  this  session. 

TTS  Slides 

Post-Execution 

Team  Decision-Making 
Debrief  Coordination 

Training  Director  assembles  0/Cs 
and  training  observers  to  identify 
problems  and  issues  for  Team 
Decision-Making  Debrief. 

Training  Director 
Guidelines 

Team  Decision-Making 
Debrief  (TTS  Session) 

Training  Director  facilitates  the 

Team  Decision-Making  Debrief 
initially.  Goal  is  to  have  Squadron 
Commander  facilitate  this  session. 

TTS  Slides 

Unit  AAR 

Squadron  Commander  facilitates 
AAR  per  unit  SOP  if  tactical  AAR 
is  desired. 

Not  Provided 
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TDX  Overview 


TDX  1:  MSssion  Analysis  and  Wargaming 


Introduction  This  TDX  provides  the  Sqdn  Commander  (Cdr)  and  staff  decision* 
making  practice  using  SC4  tools  under  experiment-like  conditions. 
The  exercise  limits  current  operation  requirements  so  that  the  battle 
staff  focus  is  on  developing  techniques  and  procedures  for  their 
decision-making  process. 


Purpose  The  material  in  this  appendix  has  two  purposes: 

•  to  describe  the  actions  of  the  Training  Director,  White  Cell,  and 
OPFOR  workstation  operators  and 

•  to  outline  procedures  before,  during,  and  after  the  TDX. 


Contents  Tlie  following  materials  are  found  in  the  TDX  1  section: 


TDX  1  Overview  and  Event  Description 

5-2*2 

TDX  1  Event  Guide 

TDX  1  OPFOR  and  BLUFOR  Guidelines 

Kua 

TDX  1  Tactical  Materials 

TDXl  Plan  Sheets 

Kail 

TDXl:  IVBs Sion  Analysis  and  Wa teaming,  Continued 


TDX  overview  Strike  Force  has  been  ordered  to  move  from  the  Grafenwohr 

airhead,  occupy  AA  COUGAR,  and  to  be  prepared  to  commence 
combat  operations  within  8  hours.  F3  troops  are  reporting  they  have 
exchanged  fire  widi  dismounted  El  Doradian  reconnaissance  forces 
along  the  intanational  border.  Joint  Task  Force  Headquartos  reports 
El  Doradian  Forces  have  ceased  electronic  transmissions,  their  mr 
force  has  stood  down,  and  ground  force  units  are  moving  from 
cantonment  areas  to  dispersal  sites.  Combat  units  in  the  border  area 
have  beai  detected  moving  slowly  toward  the  border.  El  Doradian 
government  representatives  have  requested  an  urgent  meeting  with 
international  mediators  to  protest  provocative  U.S.  military  activities. 


TDX  event 
diagram 
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TDX  Event  Description 


TDX  1:  Mission  Anaiysis  and  Wargaming  ,  Continued 


Event  This  table  provides  the  event  and  event  descriptions  for  this  exercise, 

descriptions 


1.  TDX  Overview  Unit  assembles  at  training  site.  Training  Director 


conducts  in-brief,  Pre-Action  Analysis  team 
training  session,  and  establishes  startex  time. 

2.  Battle  Force  issues  order  and  Squadron  uses  a  movement  order  provided  to  them 

INTSUM  to  move  from  an  airhead  to  AA  COUGAR. 

During  movement  the  squadron  receives  an 
INTSUM  and  an  order  from  the  BF  for  a  defensive 
operation. _ 

3.  2"<*  Squadron  occupies  assembly  Squadron  occupies  AA  COUGAR.  During 

area  occupation  squadron  receives  BF  order.  Squadron 

Commander  conducts  confirmation  brief  to  BF 
commander. _ 

4.  Commander’s  Timeout  At  a  time  determined  by  the  Squadron 

Commander,  the  exercise  is  paused  for  a 
Commander’s  Timeout  exercise.  The  Squadron 
Conunander  informs  the  staff  of  his  answers  to  a 
series  of  specific  questions  regarding  situational 
awareness  at  specific  times  or  events  during  the 
_ TDX.  _ 

5.  2"^  Squadron  Scouts  establish  Squadron  establishes  a  screen  line  in  sector  along 

screen  line  PL  TEXAS  as  directed  in  die  BF  order. _ 

6.  2’^  Squadron  develops  and  Squadron  completes  its  decision-making  process 

issues  Defense  order  and  issues  its  order  to  its  subordinate  units. 

During  the  process  the  squadron  commander  back 
briefs  his  plan  to  the  BF  commander. 

7.  Team  Decision-Making  Debrief  Training  Director  conducts  the  Team  Decision- 

Making  Debrief  and  releases  unit  to  the  Squadron 
Commander  for  tactical  AAR. 


TDXl  Event  Guide 


The  TDX 1  Event  Guide  provides  exercise  and  control  instructions  for 
Purpose  the  Training  Director,  White  Cell,  and  OPFOR. 


EVENT  2:  Battle  Force  issues  WARNO  and  INTSUM 


TDX  Workstation  Guidelines 


OPFOR  Woikstation  Execution  Guidelines 


Overview  As  the  OPFOR  workstation  operator,  you  will  need  the  following  information 

to  support  the  execution  of  the  TDX  1  exercise. 


Focus  The  purpose  of  your  actions  in  controlling  the  OPFOR  is  to  provide  the 

Squadron  with  an  opportunity  to  collect  intelligence  information  about  the 
compositions  arwi  deployment  of  the  forces  that  are  opposing  them  in  their 
area  of  interest  The  forces  available  to  you  in  this  exercise  are  as  follows: 


intent 


The  intent  of  die  OPFOR  is  move  its  forces  from  their  mardi  column 
formations  used  to  approach  the  border  area  into  more  dispersed  formations 
along  side  of  roads  which  decrease  their  vulnerability  to  air  attack  and  yet 
permit  them  to  rapidly  resume  their  approach  to  the  border. 


OPFOR  Workstation  Execution  Guidelines,  Continued 


Exercise  At  the  direction  of  the  OPFOR  Controller,  you  will  begin  to  move  your  forces 

guidance  out  of  their  column  formations  into  more  dispersed  formations  in  close 

proximity  to  the  road  network.  Attempt  to  maximize  use  of  the  available 
cover  in  the  area.  Once  this  repositioning  has  been  accomplished,  no  other 
OPFOR  activity  is  anticipated.  If  BLUFOR  UAVs  attempt  to  cross  the 
international  border,  engage  them  with  the  appropriate  weapon  systems.  Be 
prepared  to  assist  in  developing  additional  intelligence  information  about  the 
OPFOR  whidi  the  White  Cell  may  have  to  provide  in  response  to  information 
requests  from  the  experimental  unit 


Rules  of 
engagement 


Ensure  that  OTFOR  reconnaissance  patrols  do  not  cross  international  border. 
While  dispersing  units,  minimize  movement  toward  the  border  that  might  be 
interpreted  as  the  initiation  of  hostilities  by  the  experimental  unit 


Tactical  Materials 


The  Battle  Force  (BF)  tactical  materials  provide  the  cues  that  initiate 
Purpose  squadron  operations  and  the  battle  staffs  decision-making  process.  During 

the  exercise  they  are  transmitted  electronically  from  the  White  Cell  using  the 
whiteboard,  overlay  files,  or  a  combination  of  both.  Paper  copies,  including 
overlay  sketches,  are  provided  in  this  section  and  the  attachment  for 
preparation  and  reference  during  the  exercise. 


The  Brigade  Order  with  RSTA,  Effects,  and  CSS  are  located  in  the 
Materials  attachment  5-6-1.  Found  on  the  following  pages,  the  other  tactical  materials 

for  this  TDX  are: 


Sqdn  Movement  Order 

At  start  of  the  exercise 

WTiiteboard 

Movement  Overlay 

With  movement  order 

Overlay  file 

INTSUM  99-04-14-1 

When  sqdn  lead  element  reaches  CP  1 

WTiiteboard 

WARNO  99-01  (Order  #1) 

When  sqdn  lead  element  reaches  CP  3 

WTiiteboard 

WARNO  Overlay 

With  WARNO  99-01 

Overlay  file 
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TDX 1  Plan  Sheets 


T  contains  the  ModSAF  plan  sheets  that  document  locations  and  conditions  of 
Purpose  BLUFOR  and  OPFOR  entities  at  the  start  of  TDX  1 .  These  plan  sheets 
should  not  be  needed  unless  the  existing  ModSAF  files  are  damaged. 


Experimental  Unit  Data 

|2nd  Sqn,  2nd  Strike  Force 

HQ66 

QA102108 

280 

Column 

Command  1 

HC)65 

Command  2 

HQ63 

Control  1 

HQ62 

Control  2 

UAV 

HQ71 

UAV 

HQ72 

1 

MB 

HQ81 

1 

MB 

HQ82 

MB 

HQ83 

MB 

HQ84 

- 

1  E  Troop 

FV 

A66 

QA107106 

280 

Column 

FV 

A55 

D/I  Inf 

Fire  Tm  A,  F 

IQSqd 

Rides  A66  until  dismounted 

D/I  Inf 

Fire  Tm  B,  K 

[QSqd 

Rides  A55  until  dismounted 

1  1st  Pit 

FV 

All 

MAV 

A12 

i 

A16  Robotic  Follower  w/NLOS 
missiles  D 

FV 

A13 

- - 

zd 
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Team  Training  Sessions  Slides 
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Appendix  E 


Sample  Materials  from  Volume  4  of  the 
Training  and  Measurement  Support  Package 

This  appendix  provides  a  sample  set  of  the  materials  contained  Volume  4:  Measures. 
Volume  4  is  part  of  the  five  volume  set,  Training  and  Measurement  Support  Package,  Mounted 
Maneuver  Battlespace  Lab,  Battle  Command  Reengineering  III  (Training  and  Measurement 
Support  Package,  1999),  which  documents  the  work  performed  in  the  design  of  a  prototype 
training  program  and  evaluation  package. 

The  purpose  of  this  appendix  is  twofold.  First,  it  provides  the  reader  with  brief  samples 
of  the  measures  used  to  evaluate  the  training  and  staff  performance  during  BCR  HI  that  were 
mentioned  in  the  report.  Secondly,  the  appendix  provides  a  general  overview  of  the  topics  and 
materials  found  in  Volume  4. 

The  table  below  provides  a  list  of  the  materials  found  in  Volume  4  that  are  discussed  in 
this  report.  Although  surveys  and  interviews  were  designed  to  evaluate  training  as  well  as  to 
provide  feedback  on  the  BCR  HI  issues,  only  one  training  survey  and  interview  sheet  are 
provided  as  samples.  The  first  column  in  the  table  provides  the  title  for  each  product.  The 
second  column  (where  indicated)  states  whether  this  appendix  contains  a  partial  or  a  complete 
sample  of  the  product  and  the  last  column  identifies  the  page  number. 


VolumeTitle 

Appendix 

Sampk 

Appendix 

Page 

Automated  Measures 

Not  Provided 

— 

Surveys 

Partial 

E-2 

Observer’s  Data  Collection  Instrument 

Partial 

E-5 

Interviews 

Partial 
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Training  Survey  Dav  5  -  TDX  4 


Instructions: 

Earlier  this  week  you  learned  and  practiced  basic  and  advanced  skills  using  the  SC4  system. 
Today  you  participated  in  TDX4,  Brigade  Sequel.  Please  circle  the  number  (1-5)  that  best 
represents  your  response,  or  write  in  your  response,  as  appropriate,  to  the  questions  below. 

1 .  My  prior  Tools,  Functions,  and  Tasks  training  prepared  me  for  this  TDX. 

1  2  3  4  5 

Strongly  Disagree  Disagree  Neither  Disagree  Agree  Strongly  Agree 

nor  Agree 


a.  What  was  missing  from  the  training? 


b.  What  aspect  of  the  training  you  received  needs  improvement? 


2.  The  prior  TDXs  prepared  me  for  this  TDX. 

1  2  3  4  5 

Strongly  Disagree  Disagree  Neither  Disagree  Agree  Strongly  Agree 

nor  Agree 

Describe  how  the  prior  TDXs  did  or  did  not  prepare  you  for  this  TDX. 


3.  This  TDX  gave  me  a  good  chance  to  practice  using  the  SC4  system. 
1  2  3  4 

Strongly  Disagree  Disagree  Neither  Disagree  Agree 

nor  Agree 

Describe  how  this  TDX  did  or  did  not  provide  enough  practice. 


5 

Strongly  Agree 


4.  The  OPORD  and  associated  tactical  materials  gave  me  enough  information  for  planning. 

1  2  3  4  5 

Strongly  Disagree  Disagree  Neither  Disagree  Agree  Strongly  Agree 

nor^ree 

Describe  why  the  tactical  materials  did  or  did  not  give  you  enough  information. 
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TRAINING  SURVEY  DAY  5  -  TDX  4,  CONT. 


Instructions: 

The  following  statements  refer  to  three  techniques  introduced  during  training.  The  techniques 
are  defined  below.  Please  circle  the  number  (1-5)  that  best  represents  your  response  to  the  items 
below. 

Pre-Action  Analysis:  focuses  on  shared  understandings  of  roles  and  functions, 
information  management,  workload,  and  situational  awareness.  Also  helps  anticipate 
potential  problem  areas  and  generate  how  these  problems  will  be  solved,  should  they  arise. 

Conunander’s  Timeout:  recalibrates  the  team  to  the  Commander’s  situational  awareness 
by  allowing  the  Commander  to  stop  and  think  about  his  situational  awareness  and  to 
communicate  that  to  the  staff. 

Team  Decision-Making  Debrief:  discusses  the  utility  of  the  Commander’s  Timeout  and 
information  management  and  roles  and  functions  issues.  Also  identifies  action  items  for 
sustainment  and  improvement. 


Pre-Action  Analysis: 

5.  This  technique  was  useful  in  helping  us  work  together  as  a 
team. 

6.  This  technique  would  be  helpful  in  future  training. 

Commander’s  Timeout: 

7.  This  technique  was  useful  in  helping  us  work  together  as  a 
team. 

8.  This  technique  would  be  helpful  in  future  training. 

Team  Decision-Making  Debrief: 

9.  This  technique  was  useful  in  helping  us  work  together  as  a 
team. 

10.  This  technique  would  be  helpful  in  future  training. 


1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 
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TRAINING  SURVEY  DAY  5  -  TDX  4,  CONT. 


11.  What  could  we  add,  drop,  or  change  to  improve  this  week’s  training? 


12.  What  could  we  add,  drop,  or  change  to  improve  the  Pre- Action  Analysis? 


13.  What  could  we  add,  drop,  or  change  to  improve  the  Commander’s  Timeout? 


14.  What  could  we  add,  drop,  or  change  to  improve  the  Team  Decision-Making  Debrief? 
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Observer’s  Data  Collection  Instrument 


SQUADRON  COMMANDER  INTERVIEW  SHEET  -  END  OF  EXPERIMENT 


INTERVIEWER: 


DATE: 


The  first  question  asks  you  to  think  about  your  key  information.  Then  there  are  several  questions 
focused  on  the  SC4  system  capabilities.  The  third  set  of  questions  addresses  the  staff 
organization  and  multifunctional  staff  Finally,  there  are  some  questions  about  the  training. 


SC4  SYSTEM  CAPABILITIES 

1 .  Generally,  what  were  your  key  information  requirements  fon 

a.  Planning 

b.  Preparation/Rehearsal 

c.  Execution 

d.  Consolidation  and  Reorganization 

2.  Given  the  SC4  system,  was  the  required  information  accurate"} 

3...  (etc) 

TRAINING 

18.  Did  the  training  and  practice  (Levels  2-5)  during  Week  Iget  you  and  your  staff  proficient 
enough  to  use  the  SC4  system  during  the  experiment? 


19.  Did  the  training  and  practice  get  you  and  your  staff  comfortable  and  proficient  in  working 
within  the  new  staff  organization? 


20.  Did  the  training  and  practice  help  your  and  your  staff  to  understand  the  BLUFOR  and 
OPFOR  weapon  system  capabilities? 


21.  Did  you  continue  to  use  the  team  training  session  procedures  (pre-action  analysis, 

commander’s  timeout,  team  decision-making  debrief)  after  the  training  week?  Why  or  why 
not? 
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Appendix  F 


Sample  Materials  from  Volume  5  of  the 
Training  and  Measurement  Support  Package 

This  appendix  provides  a  sample  set  of  the  materials  contained  Volume  5:  Data 
Collection.  Volume  5  is  part  of  the  five  volume  set,  Training  and  Measurement  Support 
Package,  Mounted  Maneuver  Battlespace  Lab,  Battle  Command  Reengineering  III  (Training  and 
Measurement  Support  Package,  1999),  which  documents  the  work  performed  in  the  design  of  a 
prototype  training  program  and  evaluation  package. 

The  purpose  of  this  appendix  is  twofold.  First,  it  provides  the  reader  with  brief  samples 
of  the  measures  used  to  evaluate  the  training  and  staff  performance  during  BCR  HI  that  were 
mentioned  in  the  report.  Secondly,  the  appendix  provides  a  general  overview  of  the  topics  and 
materials  found  in  Volume  5. 

The  table  below  provides  a  list  of  the  materials  found  in  Volume  5  that  are  discussed  in 
this  report.  Although  surveys  and  interviews  were  designed  to  evaluate  training,  as  well  as  to 
provide  feedback  on  the  BCR  HI  issues,  only  one  training  survey  and  interview  sheet  are 
provided  as  samples.  The  first  column  in  the  table  provides  the  title  for  each  product.  The 
second  column  (where  indicated)  states  whether  this  appendix  contains  a  partial  or  a  complete 
sample  of  the  product  and  the  last  column  identifies  the  page  number. 


Volume  Tide 

Appendix 
;;  ;SampIe  ' 

Appendix 
Page  ; 

Automated  Measmes  Analysis  for  BCR  HI 

Partial 

F-2 

Training  Surveys  Analysis  for  BCR  in 

Partial 

F-3 

Process  Surveys  Analysis  for  BCR  DI 

Not  Provided 

- 

Observer’s  Data  Collection  Instrament  Analysis  for  BCR  DI 

Partial 

F-5 

Interviews  for  BCR  HI 

Partial 
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BCR  in  Automated  Measures  Data  Analysis 


MMBL  Issue:  Can  reengineered  Battle  Command  provide  the  information  and  support 
system  to  assist  the  Commander’s  decision  making  process? 

Measure  of  Performance  (MOP’)  1:  Measure  of  UAV  Mission  Effectiveness 

Operational  definition:  For  each  UAV  launch,  calculate  the  %  of  OPFOR 
vehicles  (by  type  [tank,  BFV,  APC,  Arty,  ADA,  etc.])  that  are  first  detected  by  the 
UAV  under  battalion  control.  A  possible  measurement  method  is  the  number  of 
OPFOR  vehicles  first  detected  by  UAV  missions  divided  by  the  total  number  of 
OPFOR  vehicles  detected  by  all  other  sensors  and  weapon  systems.  UAV  sensor 
capabilities  will  be  determined  by  the  parameters  established  for  the  system  prior 
to  the  start  of  the  experiment.  The  UAV  flight  path  will  be  determined  by 
experimental  unit  personnel. 

Why:  The  SC4  system  enables  the  commander  to  visualize  the  battlefield  more 
effectively.  This  should  include  the  ability  to  visualize  the  information  terrain 
and  appreciate  regions  of  the  battlefield  where  information  is  incomplete  or  non¬ 
existent.  If  they  are  being  used  effectively,  UAVs  should  be  sent  to  those  areas  of 
interest  that  are  not  already  being  covered  by  other  sensors.  If  they  are  sent  to 
areas  of  interest  already  covered  by  sensors,  then  a  case  can  be  made  that  either 
the  SC4  system  is  not  helping  the  commander  better  visualize  the  battlefield  or 
the  commander  does  not  trust  the  situational  awareness  depicted  by  the  SC4 
system. 


Data: 


Experiment  Total 

Detected  by 

Tvoe 

Exoected 

All 

MUAVtSct  PIti 

% 

UAViBnl 

% 

Combined 

% 

Tanks 

215 

126 

39 

31% 

1 

1% 

40 

32% 

IFV 

408 

336 

130 

39% 

48 

14% 

178 

53% 

APC 

92 

70 

35 

50% 

15 

21% 

50 

71% 

Arty 

264 

153 

51 

33% 

34 

22% 

85 

56% 

ADA 

24 

17 

3  n 

18% 

0 

0% 

3 

18% 

Total 

1003 

702 

258 

37% 

98 

14% 

356 

51% 

Analysis:  Overall,  the  squadron  detected  70%  of  all  OPFOR  vehicles  arrayed  against 
them.  UAV  missions  accounted  for  51%  of  the  first  detections  which  indicates  that  the 
squadron  was  employing  their  UAVs  in  a  manner  that  would  increase  their  ability  to 
visualize  the  battlefield.  Of  significance  is  that  the  Micro  UAVs  (MUAV)  which  were 
controlled  by  the  scout  platoon  first  detected  over  twice  the  number  of  OPFOR  vehicles 
than  the  squadron  level  UAV.  This  is  explained  by  a  programming  error  that  gave  the 
MUAV  the  same  detection  capability  and  a  range/endurance  that  would  allow  the  scout 
platoon  to  cover  much  more  of  the  battlefield  than  had  been  originally  intended. 
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Training  Survey  TDX  4  File  Information 


Variable  Name 

GROUP  Audience  Member  Type 

Measurement  Level:  Nominal 

Value  Label 

1  Primary  Audience 

2  Extended  Audience 

ID  ID  Number 

Measurement  Level:  Nominal 

Q 1  Prior  Training  Prepared  Me  for  This  TDX 

Measurement  Level:  Ordinal 

Value  Label 

1 .00  Strongly  Disagree 
3.00  Neither  Disagree  nor  Agree 
5.00  Strongly  Agree 

Q2  Prior  TDXs  Prepared  Me  for  This  TDX 

Measurement  Level:  Ordinal 

Value  Label 

1.00  Strongly  Disagree 
3.00  Neither  Disagree  nor  Agree 
5.00  Strongly  Agtw 

Q3  TDX  Provided  Good  Chance  to  Use  SC4  System 

Measurement  Level:  Ordinal 

Value  Label 

1.00  Strongly  Disagree 
3.00  Neither  Disagree  nor  Agree 
5.00  Strongly  Agree 

Q4  Tactical  Materials  Provided  Enough  Information 

Measurement  Level:  Ordinal 

Value  Label 

1.00  Strongly  Disagree 
3.00  Neither  Disagree  nor  Agree 
5.00  Strongly  Agree 
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Training  Survey  TDX  4  File  Information,  cont 


Q5  Pre- Action  Analysis  Useful  in  Working  as  a  Team 

Measurement  Level:  Ordinal 

Value  Label 

1.00  Strongly  Disagree 
3.00  Neither  Disagree  nor  Agree 
5.00  Strongly  Agree 

Q6  Pre-Action  Analysis  Helpful  in  Future  Training 

Measurement  Level:  Ordinal 

Value  Label 

1.00  Strongly  Disagree 
3.00  Neither  Disagree  nor  Agree 
5.00  Strongly  Agree 

Q7  Commander’s  Timeout  Useful  in  Working  as  a  Team 

Measurement  Level:  Ordinal 

Value  Label 

1.00  Strongly  Disagree 
3.00  Neither  Disagree  nor  Agree 
5.00  Strongly  Agree 

Q8  Commander’s  Timeout  Helpful  in  Future  Training 

Measurement  Level:  Ordinal 

Value  Label 

1.00  Strongly  Disagree 
3.00  Neither  Disagree  nor  Agree 
5.00  Strongly  Agree 

Q9  Team  Decision-Making  Debrief  Useful  in  Working  as  a  Team 

Measurement  Level:  Ordinal 


Value  Label 

1.00  Strongly  Disagree 
3.00  Neither  Disagree  nor  Agree 
5.00  Strongly  Agree 

Q 1 0  Team  Decision-Making  Debrief  Helpful  in  Future  Training 

Measurement  Level;  Ordinal 

Value  Label 

1.00  Strongly  Disagree 
3.00  Neither  Disagree  nor  Agree 
5.00  Strongly  Agree 
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Observer’s  Data  Collection  Instrument  FUe  Information 


Variable  Name 
NODE 

DAY 


SESSION 

QUESTION 

RATING 

B_NUM 


Measurement  Level:  Scale 

Value  Label 

1.00  Command  1 
2.00  Command  2 
3.00  Control  1 
4.00  Control  2 

Mission  Day 

Measurement  Level:  Scale 

Value  Label 

2.00  Experiment  Day  2 
3.00  Experiment  Day  3 
4.00  Experiment  Day  4 
5.00  Experiment  Day  5 
6.00  Experiment  Day  6 
7.00  Experiment  Day  7 


Measurement  Level:  Scale 

Value  Label 

1.00  Morning 
2.00  Afternoon 


Measurement  Level;  Scale 


Measurement  Level:  Scale 
Missing  Values:  999.00 

Value  Label 

1 .00  Poor  Performance 
7.00  Excellent  Performance 

Behavior  Type 
Measurement  Level:  Scale 

Value  Label 

1.00  Backup 
2.00  Communication 
3.00  Coordination 
4.00  Monitoring 
5.00  Team  Orientation 
6.00  Overall 
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Question:  Did  the  training  and  practice  (Leveis  2-5)  during  Week  1  get  you  and  your  staff  proficient  enough  to 
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